fix: apache2ctl configtest at first boot, not in the build

COVERAGE.md

@@ -9,10 +9,10 @@ acceptance test of a recipe, docs/org-plan.md section 1).
 | File | Test | Lines | Note |
 | --- | --- | --- | --- |
 | `overlay/usr/lib/inithooks/lib/wordpress.sh` | `tests/wordpress.bats` (40 tests) | 99.00 percent (99/100) under kcov | every function and every branch |
-| `overlay/usr/lib/inithooks/firstboot.d/40wordpress` | `tests/hook.bats` (29 tests) | 97.73 percent (43/44) under kcov | the hook itself, run for real |
+| `overlay/usr/lib/inithooks/firstboot.d/40wordpress` | `tests/hook.bats` (31 tests) | 97.83 percent (45/46) under kcov | the hook itself, run for real, including apache2ctl configtest before the restart and a failing configtest that is fatal to the hook |
 | `overlay/usr/local/bin/keel-wp` | `tests/wrappers.bats` (25 tests) | 100 percent (8/8) under kcov | both cache branches, the quoting, the exit code it hands back, `DEBUG`, and the `turnkey-wp` link run for real |
 | `overlay/usr/local/sbin/keel-wordpress-update` | `tests/wrappers.bats` (the same 25) | 100 percent (21/21) under kcov | both guards refused and satisfied, each wp-cli call made to fail, the whole ownership boundary, and the two names it must not take from the environment |
-| `tests/lib/boot-test-lib.sh` | `tests/boot-test.bats` (73 tests) | 98.95 percent (282/285) under kcov | parsing, addresses, deadlines, the container marks, every verdict, and the image carrying none of the build time archive files |
+| `tests/lib/boot-test-lib.sh` | `tests/boot-test.bats` (85 tests) | 99.02 percent (304/307) under kcov | parsing, addresses, deadlines, the container marks, every verdict, the image carrying none of the build time archive files and none of the certificates and keys common removes, and configtest passing in the inithooks log |
 | `conf.d/zzz-keel-archive` | `tests/keel-archive.bats` (13 tests) | 100 percent (26/26) under kcov | every way it enables and every way it refuses, including a staging keyring left in the image |
 | `conf.d/zz-project-packages` | `tests/project-packages.bats` (14 tests) | 100 percent (31/31) under kcov | shared with keel-nodebb, where the pattern is maintained |
 | `bin/keel-archive-check` | `tests/archive-check.bats` (27 tests) | 100 percent (54/54) under kcov | the build time check of tracker#7: the copy is the live archive, the entry names the keyring through signed-by, nothing says trusted=yes, and the copied InRelease verifies against the staging key |
@@ -21,7 +21,7 @@ acceptance test of a recipe, docs/org-plan.md section 1).
 | `conf.d/main` | the build | integration only | build time script, 0004 pragmatic limits |
 | `tests/boot-test.sh` | itself | integration only | the thin main of the acceptance test: keel and LXC as root |
 
-Total over the eight measured shell files: **99.12 percent (564/569)**, 221
+Total over the eight measured shell files: **99.16 percent (588/593)**, 238
 bats tests, none failing. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which the workflow
 sets to **97**, the lowest measured file. It is only ever raised (decision
 0006).
@@ -33,8 +33,8 @@ sets to **97**, the lowest measured file. It is only ever raised (decision
      100.00  31/31  zz-project-packages
      100.00  26/26  zzz-keel-archive
       99.00  99/100  wordpress.sh
-      97.73  43/44  40wordpress
-      98.95  282/285  boot-test-lib.sh
+      97.83  45/46  40wordpress
+      99.02  304/307  boot-test-lib.sh
      100.00  54/54  keel-archive-check
 
 ### The two operator commands, and the link beside each

changelog

@@ -1,3 +1,23 @@
+turnkey-wordpress-19.0 (6) turnkey; urgency=medium
+
+  * The layer builds again on common 19.x. conf.d/main ran
+    apache2ctl configtest, and both the WordPress virtual host on 443 and
+    mods-available/ssl.conf name /etc/ssl/private/cert.pem, which no layer
+    carries any more: common's removelists-final takes every private key
+    out of the image and the machine makes its own at the first boot
+    (keel-core#8). The parent layer arrives without it, so the check could
+    only fail. It now runs in firstboot.d/40wordpress, after
+    15regen-sslcert has made the certificate and before Apache is
+    restarted, the way keel-nodebb runs nginx -t in its own hook; a
+    configuration that does not pass is fatal to 40wordpress, which then
+    does not restart Apache (inithooks logs the failure and runs the next
+    hook). The boot test reads the inithooks log for the configtest line.
+    conf.d/main asserts instead that it left no cert.pem or cert.key in the
+    layer, and the boot test reads the assembled image before it boots and
+    fails on any certificate or private key in it.
+
+ -- Keel Linux maintainers <admin@keellinux.org>  Wed, 30 Sep 2026 12:00:00 +0000
+
 turnkey-wordpress-19.0 (5) turnkey; urgency=medium
 
   * bin/wordpress.py draws its password boxes on the terminal.

conf.d/main

@@ -152,9 +152,20 @@ fi
 grep -Eq '^SSLCipherSuite +[A-Za-z0-9]' "$SSL_CONF" \
     || fatal "$SSL_CONF has no cipher suite"
 
-# And the behaviour rather than the settings: Apache is asked whether it would
-# start with this configuration.
-apache2ctl configtest
+# Apache is not asked here whether it would start. wordpress.conf and
+# mods-available/ssl.conf both name /etc/ssl/private/cert.pem, and on common
+# 19.x no layer carries that file: removelists-final takes every private key
+# out of the image and the machine makes its own at the first boot
+# (15regen-sslcert). The parent layer arrives without it, so a configtest here
+# can only fail, or pass on a certificate made for the purpose that would then
+# have to be removed again. The question is asked where the answer means
+# something: firstboot.d/40wordpress runs apache2ctl configtest after the
+# certificate exists and before it restarts Apache, the way keel-nodebb runs
+# nginx -t in its own hook, and the boot test proves the site answers on 443.
+# What this script can still assert is that it did not put a key back.
+for key in /etc/ssl/private/cert.pem /etc/ssl/private/cert.key; do
+    [ ! -e "$key" ] || fatal "$key is in the layer; the machine makes its own at first boot"
+done
 
 # fab-plan-resolve keeps a package the parent layer already carries at the
 # parent's version, so the two core packages are upgraded here from the build

overlay/etc/apache2/sites-available/wordpress.conf

@@ -23,9 +23,11 @@ ServerName localhost
     ErrorLog ${APACHE_LOG_DIR}/wordpress-ssl-error.log
     CustomLog ${APACHE_LOG_DIR}/wordpress-ssl-access.log combined
 
-    # The appliance terminates TLS itself with the certificate core ships and
-    # firstboot.d/15regen-sslcert replaces on the first boot. One file holds
-    # the key and the certificate, which is why both directives name it.
+    # The appliance terminates TLS itself with the certificate
+    # firstboot.d/15regen-sslcert makes on the first boot; no layer carries
+    # one, so this file names a path that exists only on a booted machine. One
+    # file holds the key and the certificate, which is why both directives
+    # name it.
     SSLEngine on
     SSLCertificateFile /etc/ssl/private/cert.pem
     SSLCertificateKeyFile /etc/ssl/private/cert.pem

overlay/usr/lib/inithooks/firstboot.d/40wordpress

@@ -141,7 +141,15 @@ WP_KEEL_USER="$admin_user" WP_KEEL_PASS="$APP_PASS" \
     || fatal "'$admin_user' cannot authenticate with the declared secrets.app_password"
 
 # The web server serves the site from now on. It is started rather than
-# assumed, because nothing before this hook needed it.
+# assumed, because nothing before this hook needed it. Its configuration is
+# checked first, here and not at build time: the certificate both virtual
+# hosts name is made by 15regen-sslcert on this machine, so this is the first
+# moment the check can pass for the right reason. A failure is fatal to this
+# hook, not to the first boot: inithooks logs it and runs the next hook. The
+# line on success is what the boot test reads in the inithooks log.
+apache2ctl configtest \
+    || fatal "the Apache configuration does not pass apache2ctl configtest"
+echo "Apache configuration passed apache2ctl configtest"
 systemctl restart "$WORDPRESS_WEB_SERVICE.service" \
     || fatal "$WORDPRESS_WEB_SERVICE did not start"
 

tests/boot-test.bats

@@ -529,6 +529,85 @@ EOF
     [ "$(grep -c 'still carries the build time' <<< "$output")" -eq 3 ]
 }
 
+@test "shared_keys_verdict passes on an image that carries no certificate or key" {
+    mkdir -p "$S/rootfs/etc/ssl/private" "$S/rootfs/etc/ssh"
+    install -D /dev/null "$S/rootfs/etc/ssh/sshd_config"
+    run bt_shared_keys_verdict "$S/rootfs"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"no certificate or private key in the image"* ]]
+}
+
+@test "shared_keys_verdict passes public trust stores and client config beside the keys" {
+    install -D /dev/null "$S/rootfs/etc/ssl/certs/ca-certificates.crt"
+    install -D /dev/null "$S/rootfs/etc/ssh/ssh_config"
+    install -D /dev/null "$S/rootfs/etc/ssh/sshd_config"
+    run bt_shared_keys_verdict "$S/rootfs"
+    [ "$status" -eq 0 ]
+    [[ "$output" != *"the image carries"* ]]
+}
+
+@test "shared_keys_verdict names a dangling symlink where a key would be" {
+    mkdir -p "$S/rootfs/etc/ssl/private"
+    ln -s /nonexistent/cert.pem "$S/rootfs/etc/ssl/private/cert.pem"
+    run bt_shared_keys_verdict "$S/rootfs"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"the image carries /etc/ssl/private/cert.pem"* ]]
+}
+
+@test "configtest_verdict passes when 40wordpress logged a passing configtest" {
+    printf 'Syntax OK\nApache configuration passed apache2ctl configtest\n' > "$S/inithooks.log"
+    run bt_configtest_verdict "$S/inithooks.log"
+    [ "$status" -eq 0 ]
+    [[ "$output" == *"ran apache2ctl configtest and it passed"* ]]
+}
+
+@test "configtest_verdict refuses a log with a failed configtest" {
+    printf 'fatal [40wordpress]: the Apache configuration does not pass apache2ctl configtest\n' > "$S/inithooks.log"
+    run bt_configtest_verdict "$S/inithooks.log"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"reports a failed apache2ctl configtest"* ]]
+}
+
+@test "configtest_verdict refuses a log where configtest never ran" {
+    printf 'Syntax OK\n' > "$S/inithooks.log"
+    run bt_configtest_verdict "$S/inithooks.log"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"does not show apache2ctl configtest passing"* ]]
+}
+
+@test "configtest_verdict refuses a missing log" {
+    run bt_configtest_verdict "$S/no-such.log"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"no inithooks log"* ]]
+}
+
+@test "shared_keys_verdict names the cert.pem Apache reads" {
+    install -D /dev/null "$S/rootfs/etc/ssl/private/cert.pem"
+    run bt_shared_keys_verdict "$S/rootfs"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"the image carries /etc/ssl/private/cert.pem"* ]]
+    [[ "$output" != *"no certificate or private key"* ]]
+}
+
+@test "shared_keys_verdict matches the SSH host keys through the glob" {
+    install -D /dev/null "$S/rootfs/etc/ssh/ssh_host_ed25519_key"
+    install -D /dev/null "$S/rootfs/etc/ssh/ssh_host_ed25519_key.pub"
+    run bt_shared_keys_verdict "$S/rootfs"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"/etc/ssh/ssh_host_ed25519_key,"* ]]
+    [[ "$output" == *"/etc/ssh/ssh_host_ed25519_key.pub,"* ]]
+}
+
+@test "shared_keys_verdict reports every key, not only the first" {
+    install -D /dev/null "$S/rootfs/etc/ssl/private/cert.pem"
+    install -D /dev/null "$S/rootfs/etc/ssl/private/cert.key"
+    install -D /dev/null "$S/rootfs/etc/webmin/miniserv.pem"
+    install -D /dev/null "$S/rootfs/usr/share/turnkey-ssl/cert.pem"
+    run bt_shared_keys_verdict "$S/rootfs"
+    [ "$status" -eq 1 ]
+    [ "$(grep -c 'a key every machine would share' <<< "$output")" -eq 4 ]
+}
+
 @test "sources_verdict refuses a disabled source, a staging suite, another archive and another keyring" {
     _sources https://archive.keellinux.org trixie no
     run bt_sources_verdict "$S/keel.sources"

tests/boot-test.sh

@@ -27,6 +27,11 @@
 #      has is downloaded again and put through dpkg
 #   8. keel diff reports no drift between the spec and the machine
 #
+# And before any of that, on the assembled tree before it boots: the image
+# carries no certificate or private key, since the machine makes its own. And
+# right after the first boot: the inithooks log shows 40wordpress ran
+# apache2ctl configtest and it passed.
+#
 # Called by the reusable workflow test-appliance.yml after keel pull and keel
 # verify; runnable by hand as root on any host with LXC, see tests/README.md.
 # It builds nothing: the layers come from the mirror or from a directory
@@ -79,6 +84,8 @@ rm -rf "$container_dir"
 mkdir -p "$BT_ROOTFS"
 keel pull "$BT_APPLIANCE" --source "$BT_LAYERS_DIR" --cache-dir "$BT_CACHE_DIR" --non-interactive
 keel assemble "$BT_APPLIANCE" --rootfs "$BT_ROOTFS" --cache-dir "$BT_CACHE_DIR" --non-interactive
+# The image as published, before the first boot makes this machine's own keys.
+bt_shared_keys_verdict "$BT_ROOTFS"
 
 # 2. The container marks, the instance spec, the secrets it references and the
 #    conf the first boot hooks read. bt_mark_container does what buildtasks'
@@ -133,6 +140,7 @@ first_boot_done() {
 bt_wait_for "$BT_TIMEOUT" "$BT_INTERVAL" "the first boot of $BT_NAME to finish" \
     first_boot_done
 log "first boot finished; ssh root@$addr"
+bt_configtest_verdict "$BT_ROOTFS/var/log/inithooks.log"
 
 # 6. The site answers on both ports. WordPress is up as soon as Apache is, but
 #    40wordpress restarts Apache at the end of the first boot, so the page is

tests/hook.bats

@@ -2,8 +2,8 @@
 # Unit tests of overlay/usr/lib/inithooks/firstboot.d/40wordpress, the first
 # boot hook, executed for real against scratch directories.
 #
-# Everything it touches is a stub first in PATH (systemctl, mysqladmin, mysql,
-# php, wp, chown, openssl) writing a line per call into a log the tests read,
+# Everything it touches is a stub first in PATH (systemctl, apache2ctl,
+# mysqladmin, mysql, php, wp, chown, openssl) writing a line per call into a log the tests read,
 # and INITHOOKS_PATH is a scratch tree whose lib is a symlink to the real
 # library, so kcov measures the file the appliance ships. No test needs root, a
 # database, a web server or a network.
@@ -45,6 +45,7 @@ EOF
     STUBS="$SCRATCH/bin"
     mkdir -p "$STUBS"
     _stub systemctl 0
+    _stub apache2ctl "\${WP_TEST_CONFIGTEST_RC:-0}"
     _stub chown 0
     _stub mysqladmin 0
     # openssl is only used for the throwaway password
@@ -184,6 +185,26 @@ EOF
     grep -q "systemctl restart apache2.service" "$CALLS"
 }
 
+@test "the Apache configuration is tested before the web server is restarted" {
+    run bash "$HOOK"
+    [ "$status" -eq 0 ]
+    configtest=$(grep -n "^apache2ctl configtest$" "$CALLS" | cut -d: -f1)
+    restart=$(grep -n "^systemctl restart apache2.service$" "$CALLS" | cut -d: -f1)
+    [ -n "$configtest" ]
+    [ -n "$restart" ]
+    [ "$configtest" -lt "$restart" ]
+    [[ "$output" == *"Apache configuration passed apache2ctl configtest"* ]]
+}
+
+@test "a configuration that fails configtest is fatal and Apache is not restarted" {
+    export WP_TEST_CONFIGTEST_RC=1
+    run bash "$HOOK"
+    [ "$status" -ne 0 ]
+    [[ "$output" == *"does not pass apache2ctl configtest"* ]]
+    grep -q "^apache2ctl configtest$" "$CALLS"
+    run ! grep -q "systemctl restart apache2.service" "$CALLS"
+}
+
 @test "the rendered wp-config.php is checked as PHP before it is moved into place" {
     run bash "$HOOK"
     [ "$status" -eq 0 ]

tests/lib/boot-test-lib.sh

@@ -54,6 +54,21 @@ BT_SOURCES="etc/apt/sources.list.d/keel.sources"
 BT_BUILD_LEFTOVERS="srv/keel-apt
 etc/apt/sources.list.d/keel-staging.list
 etc/apt/keyrings/keel-staging-keyring.asc"
+# The certificates and private keys no layer may carry, relative to the rootfs
+# and globbed: the list common/removelists-final/turnkey removes (keel-core#8).
+# Every machine built from a layer holds the same bytes, so a key left in one
+# is a key every machine shares; the machine makes its own at the first boot.
+# Read on the assembled tree before it boots, because after the first boot
+# every one of them exists again, made on the machine.
+BT_SHARED_KEYS="etc/ssl/private/cert.pem
+etc/ssl/private/cert.key
+etc/ssl/private/ssl-cert-snakeoil.key
+etc/ssl/certs/ssl-cert-snakeoil.pem
+etc/webmin/miniserv.pem
+usr/share/turnkey-ssl/cert.pem
+usr/share/turnkey-ssl/cert.key
+etc/ssh/ssh_host_*_key
+etc/ssh/ssh_host_*_key.pub"
 # What the two update proofs use, beyond apt-get update itself.
 #
 # A project package the image already carries, to show that apt would take its
@@ -547,6 +562,44 @@ bt_build_leftovers_verdict() {
     echo "boot-test: no build time package source, archive copy or staging keyring in the image"
 }
 
+bt_shared_keys_verdict() {
+    # bt_shared_keys_verdict ROOTFS: the assembled image, not yet booted,
+    # carries no certificate or private key. conf.d/main no longer runs
+    # apache2ctl configtest because of exactly this, so the absence is what
+    # the test asserts, and every file found is named.
+    local rootfs=$1 pattern path found=0
+    while read -r pattern; do
+        [ -n "$pattern" ] || continue
+        for path in "$rootfs"/$pattern; do
+            [ -e "$path" ] || [ -L "$path" ] || continue
+            echo "boot-test: the image carries /${path#"$rootfs"/}, a key every machine would share" >&2
+            found=1
+        done
+    done <<< "$BT_SHARED_KEYS"
+    [ "$found" -eq 0 ] || return 1
+    echo "boot-test: no certificate or private key in the image"
+}
+
+bt_configtest_verdict() {
+    # bt_configtest_verdict LOG: 40wordpress ran apache2ctl configtest on the
+    # booted machine and it passed. conf.d/main no longer runs it, so the
+    # inithooks log is the only record that the check happened at all.
+    local log=$1
+    if [ ! -r "$log" ]; then
+        echo "boot-test: no inithooks log at $log" >&2
+        return 1
+    fi
+    if grep -q 'does not pass apache2ctl configtest' "$log"; then
+        echo "boot-test: 40wordpress reports a failed apache2ctl configtest" >&2
+        return 1
+    fi
+    if ! grep -q 'Apache configuration passed apache2ctl configtest' "$log"; then
+        echo "boot-test: the inithooks log does not show apache2ctl configtest passing" >&2
+        return 1
+    fi
+    echo "boot-test: 40wordpress ran apache2ctl configtest and it passed"
+}
+
 bt_sources_verdict() {
     # bt_sources_verdict FILE: the appliance's own APT source, as it ships:
     # enabled, the signed distribution, our keyring, and never a staging one.