Fixes #XXXXX - Add enhanced Apache log format and route Rails logs to journald - #22
Open
pablomh wants to merge 15 commits into
Open
Fixes #XXXXX - Add enhanced Apache log format and route Rails logs to journald#22pablomh wants to merge 15 commits into
pablomh wants to merge 15 commits into
Conversation
Introduces a new role that creates and configures the foremanctl system user for running Podman Quadlet services in rootless mode: - Creates system user/group with configurable home, shell, subuid/subgid - Configures unprivileged port binding via sysctl - Enables systemd lingering for the user - Creates user-scope Quadlet and systemd directories - Updates check_subuid_subgid to use the new role variables - Adds acl package to pre_install (required for Ansible become_user) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- deploy.yaml: add rootless_user role setup play, map XDG_RUNTIME_DIR - vars/base.yaml: add foremanctl user/group/home/quadlet dir variables - vars/default_certificates.yml: use foremanctl_home for cert directory - vars/installer_certificates.yml: update cert directory to foremanctl_home - systemd_target: run as rootless user with scope=user Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- certificates: set owner/group to foremanctl_user on all generated cert files and directories so they can be read by the rootless user - checks: use systemctl --machine=foremanctl@ --user to inspect user-scope services; check linger and unprivileged port settings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Run the PostgreSQL Podman Quadlet as the foremanctl rootless user: - Wrap all container operations in become_user: foremanctl block - Use podman unshare to set container UID/GID ownership on data dir - Add scope: user to systemd tasks and handlers - Add container UID/GID defaults - Slurp SSL cert/key as root before become_user block to avoid permission denied when creating podman secrets from /root/ paths - Add postgresql.container.d/ssl.conf Quadlet override to mount SSL secrets into the container - Add REGISTRY_AUTH_FILE to image pull Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Run the Redis Podman Quadlet as the foremanctl rootless user: - Wrap container operations in become_user: foremanctl block - Use podman unshare to set container UID/GID ownership on data dir - Add scope: user to systemd tasks and handler - Add container UID/GID and data dir defaults - Add REGISTRY_AUTH_FILE to image pull Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Run Candlepin Podman Quadlet as the foremanctl rootless user: - Wrap all container/secret operations in become_user: foremanctl block - Add scope: user to systemd tasks and handler - Move log directory creation outside become_user (runs as root) - Add REGISTRY_AUTH_FILE to image pull - Add candlepin-db-ca secret for external database SSL CA cert - Mount candlepin-db-ca in the container at candlepin_database_ssl_ca_path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Run Pulp Podman Quadlets as the foremanctl rootless user: - Wrap all container/secret operations in become_user: foremanctl block - Use podman unshare to set container UID/GID ownership on /var/lib/pulp - Add scope: user to systemd tasks and handlers - Add REGISTRY_AUTH_FILE to all three image pulls (api, content, worker) - Add pulp-db-ca secret for external database SSL CA cert, mount in all containers including migration and admin-password one-shot containers - Add After=foreman.target to all Pulp service unit files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Run Foreman and Dynflow Podman Quadlets as the foremanctl rootless user: - Wrap all container/secret operations in become_user: foremanctl block - Add scope: user to systemd tasks and handlers - Add foreman-db-ca secret for external database SSL CA cert; use foreman_database_ssl_ca_path (container path) in DATABASE_URL - Mount foreman-db-ca in all containers (foreman, dynflow, db-migrate) - Add After=foreman.target ordering to Foreman and Dynflow units - Flush handlers after proxy registration so Refresh Foreman Proxy fires after the proxy is already registered Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Run Foreman Proxy Podman Quadlet as the foremanctl rootless user: - Wrap all container/secret operations in become_user: foremanctl block - Add scope: user to systemd tasks and handler - Add Refresh Foreman Proxy handler for post-registration proxy refresh - Move feature loops inside become_user block; create container.d dir in foremanctl_quadlet_dir instead of /etc/containers/systemd - Adapt feature.yaml to use foremanctl_quadlet_dir and foremanctl owner/group for container.d override files - Adapt remote_execution_ssh feature task: generate SSH key in foremanctl_home instead of /root - Flush handlers after proxy registration (not before) - Add fapolicyd workaround to CI for Ansible become_user with foremanctl Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Pass --user=foremanctl to forge test so tests check user-scope systemd services instead of root Podman containers - Use systemctl --machine=foremanctl@ --user to stop foreman.target in the upgrade job (it is now a user-scope target) - Add fapolicyd rule to allow Ansible become_user with foremanctl: Python module files placed in /var/tmp/ need to be executable Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a --user pytest option so the test suite can check services in either rootful (Podman containers) or rootless (user-scope systemd) mode without code duplication: - conftest.py: add GenericService abstraction that checks either podman containers or user-scope systemctl depending on the --user option; add user, user_uid, user_service and database_user_service fixtures; fix certificates fixture to read ca_directory from the template - target_lifecycle_test.py: use --machine=foremanctl@ --user for systemctl stop/start/restart; extend timeout for external DB mode - All service tests (foreman, candlepin, pulp, redis, postgresql, foreman_proxy, foreman_target): use user_service fixture to check services via user-scope systemd when --user=foremanctl is passed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- remote-database: add rootless_user role setup play so the database host also runs PostgreSQL as a rootless user - deploy-dev: add rootless_user role setup for the development env - docs/certificates.md: document that certificate files are now owned by foremanctl instead of root Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Provides a migration path for existing rootful deployments to the new rootless Podman Quadlet setup. Handles stopping root-level services, transferring data directory ownership, and starting the new user-scope services. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the hardcoded 'combined' log format in both HTTP and HTTPS
vhosts with a configurable httpd_access_log_format variable defaulting
to an enhanced format:
%h %l %u %t "%r" %>s %b %D "%{X-Forwarded-For}i" "%{User-agent}i"
Two additions over standard combined:
- %D: request time in microseconds. Exposes proxy/SSL overhead that
Rails never measures and reveals client-visible timeouts (requests
with 504/408 in Apache that have no matching Rails Completed line).
- %{X-Forwarded-For}i: preserves the original client IP chain. Without
this Satellite Apache only sees the capsule IP, not the host IP,
making it impossible to trace which host triggered which registration.
Set httpd_access_log_format: combined to revert to Apache's default.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Foreman (Puma) and Dynflow sidekiq both run inside Podman containers whose log directories are not mounted to the host. This means production.log and any structured registration log entries are invisible to operators and to log analysis tooling that reads /var/log/foreman/production.log. Adding RAILS_LOG_TO_STDOUT=true redirects all Rails logging through the container stdout, which systemd/journald captures automatically. Logs become accessible via: journalctl --user -u foreman podman logs foreman Candlepin already exposes its logs via volume mount; this brings Foreman to the same level of host observability without adding a persistent log directory on the host. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two observability improvements for foremanctl deployments:
1. Enhanced Apache log format (
foreman_combined)Adds
httpd_access_log_formatvariable to thehttpdrole with theforeman_combinedformat as default:Both
foreman-vhost.conf.j2andforeman-ssl-vhost.conf.j2use{{ httpd_access_log_format }}instead of the hardcodedcombinedformat. Operators can override per-deployment via inventory or group_vars.%D— end-to-end request time at the Apache layer (microseconds); exposes proxy overhead not visible in Rails logs%{X-Forwarded-For}i— client IP chain through proxy hopsTestinfra tests added to verify
foreman_combinedis defined in both vhost configs.2. Rails logs routed to journald (
RAILS_LOG_TO_STDOUT)Adds
RAILS_LOG_TO_STDOUT: "true"to the Foreman (Puma) and Dynflow container environments insrc/roles/foreman/tasks/main.yaml.Why this is needed: The Foreman container only mounts
foreman-data-run:/var/run/foreman:z— there is no log volume.production.logis written inside the container at/usr/src/app/log/production.logand is completely inaccessible from the host. WithRAILS_LOG_TO_STDOUT, all Rails logging (including the:registrationnamed logger) goes to stdout, which systemd/journald captures.Logs become accessible via:
registration_metrics.pydetects the absence of/var/log/foreman/production.logand falls back tojournalctl -u foreman --output=cat --no-pagerautomatically when using--inventorymode.Companion PRs
registration-observabilityregistration-observabilityregistration-observability:registrationnamed logger (routed to journald by this PR)Test plan
pytest tests/httpd_test.pyforeman_combineddefined in/etc/httpd/conf.d/foreman.confforeman_combineddefined in/etc/httpd/conf.d/foreman-ssl.confRAILS_LOG_TO_STDOUT=truein Foreman container:podman inspect foreman | grep RAILS_LOGjournalctl -u foremanshows Rails log output after container startregistration_metrics.py -i inventory.inifalls back to journalctl whenproduction.logis absent🤖 Generated with Claude Code