Skip to content

Use a unix socket between httpd and Foreman's Puma - #593

Closed
ekohl wants to merge 2 commits into
theforeman:masterfrom
ekohl:foreman-unix-socket
Closed

Use a unix socket between httpd and Foreman's Puma#593
ekohl wants to merge 2 commits into
theforeman:masterfrom
ekohl:foreman-unix-socket

Conversation

@ekohl

@ekohl ekohl commented Jun 23, 2026

Copy link
Copy Markdown
Member

Why are you introducing these changes? (Problem description, related links)

#118 included this previously, but splitting the focus.

In foreman-installer we saw using a unix socket had reliability benefits.

What are the changes introduced in this pull request?

How to test this pull request

Steps to reproduce:

  • The Foreman UI loads
  • Test suite passes

Checklist

  • Tests added/updated (if applicable)
  • Documentation updated (if applicable)

@ekohl

ekohl commented Jun 23, 2026

Copy link
Copy Markdown
Member Author

Looking at the logs I see in the DB migration:

Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68208]: 2026-06-23T18:30:34 [W|app|] Failed to register foreman_remote_execution plugin (PG::UndefinedTable: ERROR:  relation "job_invocations" does not exist
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68208]:  | LINE 9:  WHERE a.attrelid = '"job_invocations"'::regclass
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68208]:  |                             ^
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68208]:  | )
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68196]: 2026-06-23T18:30:34 [W|app|] Failed to register foreman_remote_execution plugin (PG::UndefinedTable: ERROR:  relation "job_invocations" does not exist
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68196]:  | LINE 9:  WHERE a.attrelid = '"job_invocations"'::regclass
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68196]:  |                             ^
Jun 23 18:30:34 quadlet.example.com foreman-db-migrate[68196]:  | )

And then later on service startup:

Jun 23 18:31:46 quadlet.example.com foreman[68464]: 2026-06-23T18:31:46 [W|dyn|] Error on on_execution_plan_save event
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 2026-06-23T18:31:46 [I|dyn|] Backtrace for 'Error on on_execution_plan_save event' error (ActiveRecord::SerializationFailure): PG::TRSerializationFailure: ERROR:  could not serialize access due to read/write dependencies among transactions
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | DETAIL:  Reason code: Canceled on identification as a pivot, during conflict in checking.
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | HINT:  The transaction might succeed if retried.
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | 
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `exec_query'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:132:in `exec_insert'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql/database_statements.rb:88:in `exec_insert'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:167:in `insert'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/persistence.rb:496:in `_insert_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/persistence.rb:1098:in `_create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/counter_cache.rb:166:in `_create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/locking/optimistic.rb:84:in `_create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/attribute_methods/dirty.rb:226:in `_create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/callbacks.rb:459:in `block in _create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/callbacks.rb:107:in `run_callbacks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/callbacks.rb:929:in `_run_create_callbacks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/callbacks.rb:459:in `_create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/timestamp.rb:108:in `_create_record'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/persistence.rb:1069:in `create_or_update'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/callbacks.rb:455:in `block in create_or_update'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/autosave_association.rb:370:in `around_save_collection_association'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/callbacks.rb:138:in `run_callbacks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/callbacks.rb:455:in `create_or_update'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/timestamp.rb:126:in `create_or_update'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/persistence.rb:648:in `save!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/validations.rb:53:in `save!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/transactions.rb:302:in `block in save!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/transactions.rb:302:in `save!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/suppressor.rb:54:in `save!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/task_synchronization.rb:47:in `on_execution_plan_save'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/task_synchronization.rb:32:in `sync_execution_plan_to_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/hooks.rb:69:in `block (2 levels) in run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/hooks.rb:67:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/hooks.rb:67:in `block in run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/executors.rb:18:in `run_user_code'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/hooks.rb:66:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan.rb:163:in `block (2 levels) in run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:57:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:57:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:57:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/load_setting_values.rb:30:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_request_id.rb:26:in `block in hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_request_id.rb:25:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_timezone.rb:24:in `block in hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_timezone.rb:24:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_taxonomies.rb:23:in `block in hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_taxonomies.rb:23:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:57:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_user.rb:28:in `block in hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_user.rb:48:in `restore_current_user'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_user.rb:28:in `hook'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/world.rb:31:in `execute'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan.rb:162:in `block in run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan.rb:161:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan.rb:161:in `run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan.rb:259:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/world.rb:201:in `delay_with_options'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/world.rb:195:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:61:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/models/foreman_tasks/recurring_logic.rb:78:in `trigger_repeat_after'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/models/foreman_tasks/recurring_logic.rb:70:in `start_after'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/models/foreman_tasks/recurring_logic.rb:64:in `start'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:99:in `block (2 levels) in register_scheduled_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:96:in `block in register_scheduled_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/transactions.rb:209:in `transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:90:in `register_scheduled_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks/engine.rb:174:in `block (2 levels) in <class:Engine>'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails/configuration.rb:68:in `block in run_on_init_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails/configuration.rb:68:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails/configuration.rb:68:in `run_on_init_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails.rb:41:in `block in initialize!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:kernel>:90:in `tap'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails.rb:39:in `initialize!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | config/puma/production.rb:58:in `block in _load_from'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:329:in `block in run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:323:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:323:in `run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster/worker.rb:57:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:230:in `worker'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:106:in `block in spawn_worker'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/fork_tracker.rb:20:in `block in fork'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/fork_tracker.rb:18:in `fork'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/fork_tracker.rb:18:in `fork'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:106:in `spawn_worker'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:86:in `block in spawn_workers'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:79:in `times'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:79:in `spawn_workers'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:442:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/launcher.rb:203:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/rack/handler/puma.rb:79:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/rack-2.2.23/lib/rack/server.rb:327:in `start'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands/server/server_command.rb:38:in `start'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands/server/server_command.rb:143:in `block in perform'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:kernel>:90:in `tap'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands/server/server_command.rb:134:in `perform'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/command/base.rb:87:in `perform'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/command.rb:48:in `invoke'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands.rb:18:in `<top (required)>'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/bin/rails:9:in `<main>'
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 2026-06-23T18:31:46 [E|bac|] PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  |  (ActiveRecord::StatementInvalid)
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `exec_query'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:560:in `select'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `select_all'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/query_cache.rb:110:in `select_all'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/querying.rb:54:in `_query_by_sql'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:942:in `block in exec_main_query'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:928:in `exec_main_query'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:914:in `block in exec_queries'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:908:in `exec_queries'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:695:in `load'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation.rb:250:in `records'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/relation/delegation.rb:88:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/app/registries/setting_registry.rb:183:in `load_values'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/load_setting_values.rb:36:in `reload_setting_values'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/load_setting_values.rb:9:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_request_id.rb:5:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_timezone.rb:5:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_taxonomies.rb:5:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:29:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:28:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware.rb:20:in `pass'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/lib/actions/middleware/keep_current_user.rb:5:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/stack.rb:24:in `call'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/middleware/world.rb:31:in `execute'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:329:in `block in execute_delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `block in with_error_handling'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `catch'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:491:in `with_error_handling'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/action.rb:328:in `execute_delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan/steps/plan_step.rb:41:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/execution_plan.rb:260:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/world.rb:201:in `delay_with_options'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/world.rb:195:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:61:in `delay'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/models/foreman_tasks/recurring_logic.rb:78:in `trigger_repeat_after'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/models/foreman_tasks/recurring_logic.rb:70:in `start_after'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/app/models/foreman_tasks/recurring_logic.rb:64:in `start'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:99:in `block (2 levels) in register_scheduled_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:96:in `block in register_scheduled_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activerecord-7.0.10/lib/active_record/transactions.rb:209:in `transaction'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks.rb:90:in `register_scheduled_task'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/foreman-tasks-12.2.4/lib/foreman_tasks/engine.rb:174:in `block (2 levels) in <class:Engine>'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails/configuration.rb:68:in `block in run_on_init_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails/configuration.rb:68:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails/configuration.rb:68:in `run_on_init_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails.rb:41:in `block in initialize!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:kernel>:90:in `tap'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/dynflow-2.0.1/lib/dynflow/rails.rb:39:in `initialize!'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | config/puma/production.rb:58:in `block in _load_from'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:329:in `block in run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:323:in `each'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:323:in `run_hooks'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster/worker.rb:57:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:230:in `worker'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:106:in `block in spawn_worker'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/fork_tracker.rb:20:in `block in fork'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/activesupport-7.0.10/lib/active_support/fork_tracker.rb:18:in `fork'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:106:in `spawn_worker'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:86:in `block in spawn_workers'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:79:in `times'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:79:in `spawn_workers'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/cluster.rb:442:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/puma/launcher.rb:203:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/puma-6.6.1/lib/rack/handler/puma.rb:79:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/rack-2.2.23/lib/rack/server.rb:327:in `start'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands/server/server_command.rb:38:in `start'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands/server/server_command.rb:143:in `block in perform'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:kernel>:90:in `tap'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands/server/server_command.rb:134:in `perform'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/thor-1.3.0/lib/thor/command.rb:28:in `run'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/thor-1.3.0/lib/thor/invocation.rb:127:in `invoke_command'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/thor-1.3.0/lib/thor.rb:527:in `dispatch'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/command/base.rb:87:in `perform'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/command.rb:48:in `invoke'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/gems/gems/railties-7.0.10/lib/rails/commands.rb:18:in `<top (required)>'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `require'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:149:in `require'
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | /usr/share/foreman/bin/rails:9:in `<main>'
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [21] WARNING hook before_worker_boot failed with exception (ActiveRecord::StatementInvalid) PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [1] - Worker 3 (PID: 21) booted in 0.39s, phase: 0
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [25] WARNING hook before_worker_boot failed with exception (ActiveRecord::StatementInvalid) PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [1] - Worker 4 (PID: 25) booted in 0.39s, phase: 0
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [14] WARNING hook before_worker_boot failed with exception (ActiveRecord::SerializationFailure) PG::TRSerializationFailure: ERROR:  could not serialize access due to read/write dependencies among transactions
Jun 23 18:31:46 quadlet.example.com foreman[68464]: DETAIL:  Reason code: Canceled on identification as a pivot, during conflict out checking.
Jun 23 18:31:46 quadlet.example.com foreman[68464]: HINT:  The transaction might succeed if retried.
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [1] - Worker 1 (PID: 14) booted in 0.41s, phase: 0
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [33] WARNING hook before_worker_boot failed with exception (ActiveRecord::SerializationFailure) PG::TRSerializationFailure: ERROR:  could not serialize access due to read/write dependencies among transactions
Jun 23 18:31:46 quadlet.example.com foreman[68464]: DETAIL:  Reason code: Canceled on identification as a pivot, during conflict out checking.
Jun 23 18:31:46 quadlet.example.com foreman[68464]: HINT:  The transaction might succeed if retried.
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 
Jun 23 18:31:46 quadlet.example.com foreman[68464]: [1] - Worker 5 (PID: 33) booted in 0.39s, phase: 0
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 2026-06-23T18:31:46 [W|dyn|] Error on on_execution_plan_save event
Jun 23 18:31:46 quadlet.example.com foreman[68464]: 2026-06-23T18:31:46 [I|dyn|] Backtrace for 'Error on on_execution_plan_save event' error (ActiveRecord::SerializationFailure): PG::TRSerializationFailure: ERROR:  could not serialize access due to read/write dependencies among transactions
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | DETAIL:  Reason code: Canceled on identification as a pivot, during conflict in checking.
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | HINT:  The transaction might succeed if retried.
Jun 23 18:31:46 quadlet.example.com foreman[68464]:  | 

So I wonder what's going on here and if it's related to my change.

foreman_database_ssl_ca_path: /etc/foreman/db-ca.crt

foreman_name: "{{ ansible_facts['fqdn'] }}"
foreman_listen_stream: localhost:3000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep this consistent with PROD deployment?

Suggested change
foreman_listen_stream: localhost:3000
foreman_listen_stream: /run/httpd.foreman.sock

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design here is that the role defaults remain standalone while a level higher they're connected to Apache.

Comment on lines +16 to +21
# TODO: probably not the right boolean
- name: Set daemons_enable_cluster_mode so Apache can connect to unix sockets
ansible.posix.seboolean:
name: daemons_enable_cluster_mode
state: true
persistent: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make sense to run this only when SELinux is enabled?
Also, since it worked for you, can we remove the TODO comment as part of this change?

Suggested change
# TODO: probably not the right boolean
- name: Set daemons_enable_cluster_mode so Apache can connect to unix sockets
ansible.posix.seboolean:
name: daemons_enable_cluster_mode
state: true
persistent: true
- name: Set daemons_enable_cluster_mode so Apache can connect to unix sockets
ansible.posix.seboolean:
name: daemons_enable_cluster_mode
state: true
persistent: true
when: ansible_facts['selinux']['status'] == "enabled"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this really is the correct boolean so IMHO the TODO still stands.

- mod_ssl
state: present

- name: Set httpd_can_network_connect so Apache can connect to Puma and Gunicorn

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can now say we use httpd_can_network_connect just for Gunicorn? or we can remove this entirely when we update pulp unix sockets too?

@ekohl

ekohl commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Closing in favor of #608

@ekohl ekohl closed this Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants