Skip to content

(MODULES-11990) Fix SLES 12 mod_security apache start failures in acceptance - #2659

Merged
imaqsood merged 2 commits into
mainfrom
MODULES-11990-sles12-hostname-pin
Oct 22, 2026
Merged

imaqsood merged 2 commits into
mainfrom
MODULES-11990-sles12-hostname-pin

Conversation

@imaqsood

Copy link
Copy Markdown
Contributor

Summary

Fixes the intermittent SLES 12 acceptance failure Systemd start for apache2 failed! in spec/acceptance/mod_security_spec.rb. Both changes are test-only; module code is unchanged.

  • spec/setup_acceptance_node.pp: on SLES < 15, add an /etc/hosts entry for the node. The entry is named after the FQDN, with the short hostname as an alias, so the node's own hostname resolves without DNS.
  • spec/acceptance/mod_security_spec.rb: remove the retry wrapper (it only matched restart, and retrying didn't help). If apache fails to start, the error is now re-raised with diagnostics appended: hostname and getent, /etc/hosts, resolv.conf, systemctl status, the apache journal, the tail of error.log, and listening sockets.

Additional Context

  • Root cause and the steps to reproduce.

    SLES 12 gets httpd 2.4.23 from the Leap 42.3 repo that setup_acceptance_node.pp adds. In that version, mod_unique_id (loaded by apache::mod::security) resolves the server's own hostname over IPv4 during post_config. If the lookup fails, it returns an error and apache won't start. httpd 2.4.29 removed that check.

    On the provisioned SLES 12 VM, /etc/hosts has no entry for the node, and cat-github-actions module_acceptance.yml strips the search line from /etc/resolv.conf on all targets. That leaves only nameserver 169.254.169.254, so the short hostname is resolved by a bare DNS query, which succeeds or fails from one moment to the next.

    The failure never showed in CI logs. Puppet's systemd provider attaches journalctl -n 50 --since '5 minutes ago', which on SLES 12 prints the oldest 50 lines of that window, and this error is written only to apache's error.log.

  • Thought process behind the implementation. The fix was confirmed with two SLES-12-only experiment runs (puppetcore8 + puppetcore9 nightly, full suite):

    Jobs mod_security failures AH01564 in error.log
    A: diagnostics only 10 7 in all 7 failures, none in passes
    B: diagnostics + hosts entry 20 0 (336 examples, 0 failures in every job) 0

    Every failure in A logged [unique_id:alert] (EAI 2)Name or service not known: AH01564: unable to find IPv4 address of "litmus-<id>" followed by AH00016: Configuration Failed. In all three passing A jobs, getent ahostsv4 $(hostname) succeeded before the mod_security tests and failed about 20 s later, so those jobs passed only because apache happened to start while the lookup was working. In B, the lookup succeeded in all 40 probes.

    Making the FQDN the canonical name keeps the fqdn fact, and with it the default ServerName, stable. Earlier SLES 12 logs show it flipping to the bare hostname mid-run, which caused idempotency failures such as mod_dav_svn_spec.rb:18.

    Experiment runs: A = 39605092342, 39605145388, 39605198578, 39605251239, 39605304500; B = 39616320212, 39616369394, 39616418221, 39616467904, 39616517143, 39619664972, 39619719393, 39619774766, 39619833907, 39619886378.

Related Issues (if any)

MODULES-11990 (epic MODULES-11964, restore supported-modules nightly health).

Checklist

  • 🟢 Spec tests: spec/classes/mod/security_spec.rb 282 examples, 0 failures; rubocop clean; puppet-lint warnings unchanged from main.
  • 🟢 Acceptance tests: SLES 12 experiment above (20/20 green with the change).
  • Manually verified: rspec-puppet compile of setup_acceptance_node.pp (Host present on SLES 12, absent on SLES 15). puppet apply of the host resource writes IP fqdn short.

🤖 Generated with Claude Code

imaqsood and others added 2 commits October 21, 2026 18:35
…tance

On SLES 12 the mod_security acceptance tests intermittently fail with
"Systemd start for apache2 failed!", and the log never shows why: Puppet's
systemd provider attaches `journalctl -n 50 --since '5 minutes ago'`,
which on SLES 12's systemd prints the oldest 50 lines of the window, and
errors raised after the config is parsed only reach the apache error log.

Replace the retry wrapper with one that re-raises the failure with the
hostname/resolver state, systemctl status, the apache journal and error
log, and the listening sockets appended.

Co-Authored-By: Claude <noreply@anthropic.com>
SLES 12 gets httpd 2.4.23 from the Leap 42.3 repo. Its mod_unique_id,
loaded by apache::mod::security, refuses to start apache when the node's
own hostname doesn't resolve to an IPv4 address (AH01564, followed by
AH00016 Configuration Failed); httpd 2.4.29 dropped that check.

On the provisioned SLES 12 VM, /etc/hosts has no entry for the node and
the CI workflow strips the `search` line from /etc/resolv.conf, so the
short hostname is resolved by a bare DNS query that succeeds or fails
from one moment to the next. The diagnostics run confirmed this: every
failed start logged "(EAI 2)Name or service not known: AH01564", and the
same lookup flipped from success to failure within ~20s on passing runs.

Add an /etc/hosts entry for the node, named after the FQDN with the short
hostname as an alias, so the lookup no longer depends on DNS and the fqdn
fact (and with it the default ServerName) stays stable. With the entry in
place, 20 of 20 SLES 12 acceptance jobs (puppetcore8 and puppetcore9
nightly) passed, against 3 of 10 without it.

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The narrowly scoped test-only changes have no identified blocking issues and are supported by the reported successful SLES 12 acceptance experiments.

No open issues 🎉

What changed in this PR

This PR addresses intermittent SLES 12 mod_security acceptance failures caused by unreliable hostname resolution. Changes are limited to test setup and failure diagnostics; module behavior is unchanged.

Changes:

  • Add an FQDN-based /etc/hosts entry on SLES versions below 15, with the short hostname as an alias.
  • Replace restart retries with hostname, network, and Apache diagnostics appended to start or restart failures.
File Description
spec/​setup_acceptance_node.pp Stabilizes local hostname resolution and the canonical FQDN on older SLES nodes.
spec/​acceptance/​mod_security_spec.rb Removes retries and preserves failures with additional diagnostic output.

Review effort: Balanced

@imaqsood
imaqsood merged commit 47bba2c into main Oct 22, 2026
108 of 120 checks passed
@imaqsood
imaqsood deleted the MODULES-11990-sles12-hostname-pin branch October 22, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants