Skip to content

Do not restart agent service on package changes#981

Open
Flo2388 wants to merge 1 commit into
theforeman:masterfrom
Flo2388:fix-973/adjust-restart-behaviour
Open

Do not restart agent service on package changes#981
Flo2388 wants to merge 1 commit into
theforeman:masterfrom
Flo2388:fix-973/adjust-restart-behaviour

Conversation

@Flo2388
Copy link
Copy Markdown

@Flo2388 Flo2388 commented Mar 24, 2026

Change the relationship between puppet::agent::install and puppet::agent::config/puppet::agent::service from notification (~>) to ordering-only (->). The notification caused a reload-or-restart of the agent service immediately after a package upgrade, which killed the newly started service and left it in a failed state.

When upgrading the openvox-agent package (e.g. 8.23.1 → 8.24.1 or later), the following race condition occurs:

  1. The running puppet agent (PID A) applies the package resource, spawning apt-get/dpkg as child processes.

  2. The package's postinst script calls systemctl try-restart puppet. systemd stops the service and sends SIGTERM to the main service PID.

  3. PID A and its children (apt-get, dpkg) survive as orphan processes in the cgroup because they are child processes, not the main PID tracked by systemd.

  4. The postinst script starts a new puppet service (PID B). The service is now healthy.

  5. The orphaned PID A finishes the package resource and, due to the ~> (notification) relationship, triggers reload-or-restart on the puppet service. This sends SIGHUP to PID B — the service that was just started in step 4.

  6. PID B exits, the service enters the dead state, and no puppet agent is running.

Changing ~> to -> between puppet::agent::install and puppet::agent::service preserves ordering but removes the redundant notification. The package postinst already handles the service restart during upgrades.

@Flo2388 Flo2388 force-pushed the fix-973/adjust-restart-behaviour branch 2 times, most recently from 3548892 to 5373d4b Compare May 28, 2026 13:06
@Flo2388 Flo2388 changed the title fixes #973 - Do not restart agent service on package changes Do not restart agent service on package changes May 29, 2026
@Flo2388
Copy link
Copy Markdown
Author

Flo2388 commented May 29, 2026

fixes #973

Change the relationship between puppet::agent::install and
puppet::agent::config/puppet::agent::service from notification (~>)
to ordering-only (->). The notification caused a reload-or-restart
of the agent service immediately after a package upgrade, which
killed the newly started service and left it in a failed state.

When upgrading the openvox-agent package (e.g. 8.23.1 → 8.24.1), the
following race condition occurs:

1. The running puppet agent (PID A) applies the package resource,
   spawning apt-get/dpkg as child processes.

2. The package's postinst script calls systemctl try-restart puppet.
   systemd stops the service and sends SIGTERM to the main service PID.

3. PID A and its children (apt-get, dpkg) survive as orphan processes
   in the cgroup because they are child processes, not the main PID
   tracked by systemd.

4. The postinst script starts a new puppet service (PID B). The
   service is now healthy.

5. The orphaned PID A finishes the package resource and, due to the
   ~> (notification) relationship, triggers reload-or-restart on the
   puppet service. This sends SIGHUP to PID B — the service that was
   just started in step 4.

6. PID B exits, the service enters the dead state, and no puppet
   agent is running.

Changing ~> to -> between puppet::agent::install and
puppet::agent::service preserves ordering but removes the redundant
notification. The package postinst already handles the service
restart during upgrades.
@Flo2388 Flo2388 force-pushed the fix-973/adjust-restart-behaviour branch from 5373d4b to 74b7d96 Compare May 29, 2026 12:46
@Flo2388
Copy link
Copy Markdown
Author

Flo2388 commented Jun 3, 2026

Here is the result of the test with my fork. Update runs successfully and in the status we can see, that the service is not failed, but new version is already picked up.

I tested with an Ubuntu 22.04 node.

root@test-agent-01:~# puppet agent -t
Info: Refreshing CA certificate
Info: CA certificate is unmodified, using existing CA certificate
Info: Refreshing CRL
Info: CRL is unmodified, using existing CRL
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from puppet.example.com:8140 (203.0.113.10)
Notice: Catalog compiled by puppet.example.com
Info: Caching catalog for test-agent-01.example.com
Info: Applying configuration version '1780503021'
Notice: /Stage[main]/Puppet::Agent::Install/Package[openvox-agent]/ensure: ensure changed '8.23.1-1+ubuntu22.04' to '8.27.0-1+ubuntu22.04'
Notice: Applied catalog in 11.72 seconds
root@test-agent-01:~# systemctl status puppet
● puppet.service - Puppet agent daemon provided by OpenVox
     Loaded: loaded (/lib/systemd/system/puppet.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2026-06-03 18:10:30 CEST; 12s ago
       Docs: man:puppet-agent(8)
   Main PID: 777605 (puppet)
      Tasks: 6 (limit: 4554)
     Memory: 145.0M
        CPU: 6.603s
     CGroup: /system.slice/puppet.service
             ├─777605 /opt/puppetlabs/puppet/bin/ruby /opt/puppetlabs/puppet/bin/puppet agent --no-daemonize
             └─777747 "puppet agent: applying configuration" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Jun 03 18:10:30 test-agent-01.example.com systemd[1]: puppet.service: Deactivated successfully.
Jun 03 18:10:30 test-agent-01.example.com systemd[1]: Stopped Puppet agent daemon provided by OpenVox.
Jun 03 18:10:30 test-agent-01.example.com systemd[1]: puppet.service: Consumed 27min 36.203s CPU time.
Jun 03 18:10:30 test-agent-01.example.com systemd[1]: Started Puppet agent daemon provided by OpenVox.
Jun 03 18:10:36 test-agent-01.example.com puppet-agent[777605]: Starting Puppet client version 8.27.0
Jun 03 18:10:41 test-agent-01.example.com puppet-agent[777747]: Requesting catalog from puppet.example.com:8140 (203.0.113.10)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant