Rootless iop - #5
Open
pablomh wants to merge 108 commits into
Open
Conversation
pablomh
force-pushed
the
rootless_iop
branch
20 times, most recently
from
March 3, 2026 21:27
0e26970 to
88624aa
Compare
Introduces the deploy_network role, which wraps containers.podman.podman_network to create a named bridge network with configurable properties: internal (no external routing), isolate (no cross-network packet forwarding), IPv6, subnet, gateway, and DNS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace host networking with four named Podman bridge networks: - foreman-db (internal, isolated): PostgreSQL and its clients - foreman-cache (internal, isolated): Redis and its clients - foreman-app: application layer — Foreman, Candlepin, Pulp, httpd - foreman-proxy-net: Foreman and the smart proxy Services reach each other by container DNS name instead of localhost. Per-service notes: - postgresql: also accessible via a Unix socket bind-mounted at /var/run/postgresql, used by Ansible's community.postgresql modules during deployment so no TCP port needs to be published to the host. - redis: publishes no ports; clients reach it by DNS name on foreman-cache. - candlepin: Tomcat binds to 0.0.0.0 so that other containers on the bridge network can reach it via its DNS name. - migration containers (foreman db:migrate, pulp migrations): attached to foreman-db when the database is internal; switched to foreman-app when the database is external, because foreman-db uses internal: true which removes the default gateway, making it impossible for containers on that network to reach an external database host. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The development setup runs the Foreman Rails process directly on the host rather than in a container, so it cannot resolve container bridge DNS names such as postgresql or candlepin. Keeping host networking for all services in the devel environment means every service remains reachable on localhost as before, without requiring per-service URL overrides for the Rails process. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
foreman-certs generates certificates only for localhost and the server FQDN. Now that Foreman connects to Candlepin via the bridge DNS name "candlepin", TLS hostname validation fails against those certificates. A dedicated certificate with SAN=candlepin, signed by the installer CA, is generated using the openssl CLI (consistent with the rest of the codebase). The certificate validity is set to 7300 days (20 years), matching the default used by puppet-certs (theforeman/puppet-certs manifests/init.pp $expiration parameter). The signing step runs on every deployment to ensure the certificate is always freshly dated, consistent with how foreman-certs handles the other installer certificates. The localhost certificate previously used by Candlepin's Tomcat is now unused and removed: - "localhost" dropped from certificates_hostnames — the certificates role no longer generates the cert. - localhost_key / localhost_certificate removed from default_certificates.yml and installer_certificates.yml. The healthcheck is updated to validate the new certificate instead of skipping verification with --insecure. --resolve candlepin:23443:127.0.0.1 forces the connection to the loopback so it works in both bridge networking (where other containers reach Candlepin by DNS name) and host networking (where "candlepin" would not resolve via container DNS). Tests are updated to route connectivity checks through the foreman container (which shares the same bridge network as candlepin), to verify against the new certificate, and to check candlepin_certificate expiry instead of the now-removed localhost_certificate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds docs/deployment.md with a description of the four bridge networks introduced by this series, their properties (internal, isolate), which containers are attached to each, and the rationale for the port publishing decisions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Creates a dedicated service user (foremanctl) for running Podman containers in rootless mode. The role handles all prerequisites: - acl package (required for Ansible become_user on CentOS 10) - User and group creation with a non-login shell - subuid/subgid ranges for user namespace mapping - Unprivileged port binding via sysctl - loginctl enable-linger so the user session survives across boots - XDG_RUNTIME_DIR discovery (exposed as rootless_user_xdg_runtime_dir) - Quadlet and systemd user config directories Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…plays
base.yaml gains foremanctl_user/group/home/quadlet_dir/systemd_user_dir
vars. It also introduces foremanctl_become_method and foremanctl_runtime_env
as the single place to control how all rootless become blocks operate:
- foremanctl_become_method: swap between 'sudo' (default) or
'community.general.machinectl' (or any future method) without touching
individual roles
- foremanctl_runtime_env: holds XDG_RUNTIME_DIR for sudo; set to {} when
the become method provides the session environment automatically
deploy.yaml is split into two plays so the rootless_user role can expose
foremanctl_xdg_runtime_dir as a fact before the service roles run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move the default certificate directory from /root/certificates to /var/lib/foremanctl/certificates so that the rootless containers running in foremanctl's user namespace can read them without elevated privileges. The path is hardcoded rather than using foremanctl_home because default_certificates.yml is loaded before base.yaml and base.yaml itself references certificate path vars. ca.yml: add base directory creation; set owner/group to foremanctl_user/foremanctl_group on all directories and generated files. issue.yml: set owner/group to foremanctl_user/foremanctl_group on all generated keys, CSRs and certificates. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nuser -l Remove the duplicate run_as that was shadowing the correct implementation. Revert the runuser call back to the original working form with -l flag, XDG_RUNTIME_DIR export, and single-quote escaping (escaped), as it was in commit a2c086d. XDG_RUNTIME_DIR is required so systemctl --user can locate the D-Bus session bus socket. Also revert postgresql and pulp tests to their original commands from a2c086d, avoiding the multiple-shell-layer escaping issues introduced by subsequent "fix" commits. The original psql -c style commands work correctly through testinfra's sudo wrapper. Also fix iop_fdw and iop_inventory roles to connect to PostgreSQL via Unix socket instead of TCP (login_host), matching how all other roles connect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add AddHost={{ ansible_facts['fqdn'] }}:host-gateway to dynflow-sidekiq
containers so they can resolve the server FQDN to reach the HTTPD
reverse proxy and Pulp API. Without this, Katello background workers
fail with PulpRpmClient::ApiError: Connection failed when processing
repository tasks. The Foreman web container already had this fix
(added in 975e3d8), but the dynflow workers were missed.
Skip test_postgresql_service when database_mode=external: in that
configuration postgresql runs on a separate database VM, not on
quadlet. The companion test_postgresql_missing_with_external already
asserts the correct behaviour for external mode.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Use iop-core-gateway hostname instead of localhost when curling from inside the container. The server certificate has SAN=iop-core-gateway, so curl to https://localhost:8443/ fails TLS verification, returning 000 twice (once from -w http_code, once from || echo 000). Inside the container, iop-core-gateway resolves via aardvark-dns to the container own network IP. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove || true from upgrade stop step; the stop should succeed - Group installer certificate pre_tasks in a single block with one when condition; drop redundant become: true (inherited from play) - Remove noisy comment from candlepin handler - Fix hardcoded /var/lib/foremanctl in certs_secrets.yml comments - Add blank line before Create Candlepin configuration task - Add foremanctl_registry_auth_file to base.yaml and include REGISTRY_AUTH_FILE in foremanctl_runtime_env so all podman pull tasks in foremanctl blocks get registry authentication automatically Flush handlers after service start is intentional: restarting before start would fail on first deploy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
runuser requires root and misparses --user as its own flag. Replace with sudo -u foremanctl and set XDG_RUNTIME_DIR explicitly so systemctl --user can reach the D-Bus session bus. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
rootless_iop
branch
6 times, most recently
from
March 14, 2026 22:21
dbd6e99 to
1a99c8c
Compare
Tests run last (filename sorts after redis_test.py) since stopping the target would break any subsequent service checks. This also validates the stop command used in the upgrade workflow. Use server.run with systemctl --machine=foremanctl@ --user for all systemctl operations, consistent with how non-podman commands are run in other tests. After each lifecycle operation, poll the Foreman HTTPS frontend (https://server_fqdn/api/v2/ping with CA cert) matching the pattern used in test_https_foreman_ping. On timeout, dump user service states for diagnosis. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
rootless_iop
branch
5 times, most recently
from
March 15, 2026 03:53
36923e7 to
8ea52c7
Compare
Tests run last (filename sorts after redis_test.py) since stopping the target would break any subsequent service checks. This also validates the stop command used in the upgrade workflow. Use server.run with systemctl --machine=foremanctl@ --user for all systemctl operations. After each lifecycle operation, poll the Foreman HTTPS frontend (https://server_fqdn/api/v2/ping with CA cert) matching the pattern used in test_https_foreman_ping. On timeout, dump service states, foreman.service status, container states and foreman logs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pablomh
force-pushed
the
rootless_iop
branch
4 times, most recently
from
March 15, 2026 13:02
bf0aed4 to
0dad8d8
Compare
Tests run last (filename sorts after redis_test.py) since stopping the target would break any subsequent service checks. This also validates the stop command used in the upgrade workflow. Use server.run with systemctl --machine=foremanctl@ --user for all systemctl operations. After each lifecycle operation, poll the Foreman HTTPS frontend (https://server_fqdn/api/v2/ping with CA cert) matching the pattern used in test_https_foreman_ping. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wrap the systemd restart in block/rescue to collect service status and container logs when the restart fails, then re-raise so CI still marks the run as failed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This reverts commit fcc4096.
Foreman depends on Candlepin being fully initialised before it can
serve requests reliably. Without this ordering, foreman.service starts
simultaneously with candlepin.service, competing for CPU and disk I/O
during the JVM startup. This pushes Rails loading time close to the
default TimeoutStartSec (90s) and causes intermittent failures on
restart, particularly when the system is under load.
Because candlepin.service already declares:
After=redis.service postgresql.service
the transitive ordering becomes:
postgresql --+
+--> candlepin --> foreman
redis -------+
redis.service and postgresql.service are kept as explicit dependencies
to preserve correct ordering in future non-Katello deployments that
may not include candlepin.service.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extends foreman_target_test.py with lifecycle tests intended to run at the end of the test suite. Tests that foreman.target can be cleanly stopped and restarted, and that Foreman becomes available again afterwards via the HTTPS endpoint (same curl pattern as test_https_foreman_ping in httpd_test.py).
Incorporates upstream commits: - Use candlepin.service in foreman startup ordering (f2e162d) - Add foreman.target lifecycle tests (5a2eb8b) Conflicts resolved keeping rootless implementations; remote-database playbook updated to include both SSL cert vars and postgresql_networks. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.