Give the boot test container a systemd the first boot can rely on

COVERAGE.md

@@ -10,21 +10,21 @@ LXC as the acceptance test of a recipe, docs/org-plan.md section 1).
 | --- | --- | --- | --- |
 | overlay/usr/lib/inithooks/lib/postgresql.sh | tests/postgresql.bats (11 tests) | 100 percent (41/41) under kcov | every function and every branch |
 | overlay/usr/lib/inithooks/firstboot.d/36pgsqlverify | tests/hook.bats (10 tests) | 100 percent (15/15) under kcov | every path, including the two failures that matter |
-| tests/lib/boot-test-lib.sh | tests/boot-test.bats (37 tests) | 100 percent (141/141) under kcov | argument parsing, address discovery, deadlines, the database, module, Webmin and diff verdicts |
+| tests/lib/boot-test-lib.sh | tests/boot-test.bats (44 tests) | 100 percent (154/154) under kcov | argument parsing, address discovery, deadlines, the container marks, the database, module, Webmin and diff verdicts |
 | conf.d/main | the build | integration only | build time script, 0004 pragmatic limits |
 | tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root |
 | overlay ... firstboot.d/35pgsqlpass | common | not this repository | the hook that sets the password belongs to the `common` fork and is used, not rewritten |
 
-Total over the three measured shell files: **100 percent (197/197)**,
-58 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which
+Total over the three measured shell files: **100 percent (210/210)**,
+65 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which
 the workflow sets to 100, the measured number. It is only ever raised
 (decision 0006).
 
     $ COVERAGE_THRESHOLD=100 tests/coverage.sh
     kcov line coverage (threshold 100 percent):
      100.00  41/41  postgresql.sh
      100.00  15/15  36pgsqlverify
-     100.00  141/141  boot-test-lib.sh
+     100.00  154/154  boot-test-lib.sh
 
 ## What the hook tests cover
 
@@ -52,9 +52,49 @@ the published layer from `https://mirror.keellinux.org/layers`, verifies
 it, assembles it, boots it in LXC and runs `tests/boot-test.sh`. Nothing is
 built there.
 
-State on 2026-09-27: **the layer is not published yet, so the job skips
-with a notice and passes.** It becomes a required status on `main` the day
-the first `postgresql` layer reaches the mirror.
+### What the gate found once the layer booted (2026-09-27)
+
+The `postgresql` layer was published (104,170,947 bytes, parent `core`
+7acf2c53) and the job ran for real. It failed, on two defects at once,
+one in the test and one in the layer:
+
+    INFO: [35pgsqlpass] successfully completed
+    ERR: [36pgsqlverify] failed - exit code 1
+    psql: error: connection to server at "::1", port 5432 failed: Connection refused
+    ERR: [15regen-sslcert] failed - exit code 1
+    ERR: [95secupdates] failed - exit code 1
+
+Reproduced on the build host from the published chain.
+
+1. **The test.** Under the stock LXC container apparmor profile systemd
+   cannot give a unit a mount namespace, so `systemd-journald`,
+   `systemd-logind`, `systemd-sysusers`, `systemd-sysctl` and
+   `tmp.mount` failed with `status=226/NAMESPACE`, which is why
+   `15regen-sslcert` and `95secupdates` failed and why the container had
+   no journal. `bt_lxc_config` now writes
+   `lxc.apparmor.profile = generated` and
+   `lxc.apparmor.allow_nesting = 1`, and `bt_mark_container` does what
+   buildtasks' container patch does, both ported from keel-nodebb.
+2. **The layer**, and the hook was right to report it. The cluster was
+   online and listening on `127.0.0.1:5432` alone: Debian's `/etc/hosts`
+   maps `::1` to `ip6-localhost` and never to `localhost`, so
+   `listen_addresses = 'localhost'` binds the IPv4 loopback only. Fixed
+   by naming both addresses, in the changelog as version 2.
+
+Measured on the build host with both in place: every hook from
+`01ipconfig` to `98finalize` completes, the only failed unit is
+`inithooks-restart-getty1.service`, which needs a tty no container has,
+and
+
+    $ psql --username=postgres --host=::1 --port=5432 --no-password \
+        --command='SELECT 1, current_user, inet_server_addr()'
+    1|postgres|::1
+
+with the declared password in `PGPASSWORD`, while the wrong password
+gives `FATAL:  password authentication failed for user "postgres"`.
+
+State on 2026-09-27: green once this and the layer rebuild land. It
+becomes a required status on `main` then.
 
 ## Plan
 

tests/README.md

@@ -66,9 +66,16 @@ What it does, in order:
 
 1. `keel pull` and `keel assemble` the chain (core, postgresql) into
    `<lxc-path>/<name>/rootfs`.
-2. Creates `var/lib/turnkey-info/inithooks.service/lxc` in the rootfs, the
-   marker `bt-container` writes and the one `keel inspect` reads to call
-   the machine a container (`network.managed_by: host`).
+2. Marks the tree as a container build, which is what `bt_mark_container`
+   does and what buildtasks' `patches/container/conf` does for a real
+   container image: the marker
+   `var/lib/turnkey-info/inithooks.service/lxc` that `keel inspect` reads
+   to call the machine a container (`network.managed_by: host`),
+   `REDIRECT_OUTPUT=true` in `etc/default/inithooks`, and a drop-in giving
+   `inithooks.service` `StandardOutput=journal`. Without the last two the
+   hooks write to `/dev/tty1`, which nobody reads in a container, and the
+   first hook that prints more than the terminal buffer holds blocks there
+   forever.
 3. Writes a random `root_password` and `db_password` under
    `etc/keel/secrets` (mode 0600) and installs `tests/instance.yaml` at
    `etc/keel/instance.yaml` and `etc/inithooks.yaml`.
@@ -78,7 +85,11 @@ What it does, in order:
    would have nothing declared and no terminal, and `36pgsqlverify`
    would fail naming the field the description has to declare.
 5. Writes an LXC config for that rootfs on the bridge and starts the
-   container.
+   container. The config asks for `lxc.apparmor.profile = generated` and
+   `lxc.apparmor.allow_nesting = 1`: without them systemd cannot give a
+   unit a mount namespace, so `systemd-journald`, `systemd-logind` and
+   `tmp.mount` fail with `status=226/NAMESPACE` and the hooks that need
+   them fail beside the database.
 6. Waits for a global IPv6 address (`lxc-info -i`), then for the first boot
    to finish: `RUN_FIRSTBOOT=false` in the rootfs copy of
    `/etc/default/inithooks`, and then confconsole or an SSH banner.

tests/boot-test.bats

@@ -242,6 +242,78 @@ never() { return 1; }
     [[ $output == *"lxc.net.0.type = veth"* ]]
 }
 
+@test "lxc_config: the apparmor pair a container running systemd needs" {
+    output=$(bt_lxc_config keel-postgresql-boot-test /r/rootfs br0)
+    [[ $output == *"lxc.apparmor.profile = generated"* ]]
+    [[ $output == *"lxc.apparmor.allow_nesting = 1"* ]]
+}
+
+fake_rootfs() {
+    # fake_rootfs [VALUE]: a scratch rootfs with an inithooks defaults file,
+    # REDIRECT_OUTPUT set to VALUE (default false), printed on stdout
+    local rootfs="$BATS_TEST_TMPDIR/rootfs-$RANDOM"
+    mkdir -p "$rootfs/etc/default"
+    cat > "$rootfs/$BT_INITHOOKS_DEFAULT" <<DEF
+INITHOOKS_CONF=/etc/inithooks.conf
+RUN_FIRSTBOOT=true
+REDIRECT_OUTPUT=${1-false}
+SUDOADMIN=false
+DEF
+    printf '%s\n' "$rootfs"
+}
+
+@test "mark_container: writes the marker the unit conditions and inspect read" {
+    rootfs=$(fake_rootfs)
+    run bt_mark_container "$rootfs"
+    [ "$status" -eq 0 ]
+    [ -f "$rootfs/var/lib/turnkey-info/inithooks.service/lxc" ]
+}
+
+@test "mark_container: turns REDIRECT_OUTPUT on, so no hook blocks writing to tty1" {
+    rootfs=$(fake_rootfs false)
+    run bt_mark_container "$rootfs"
+    [ "$status" -eq 0 ]
+    grep -q '^REDIRECT_OUTPUT=true$' "$rootfs/$BT_INITHOOKS_DEFAULT"
+    # the rest of the file is left alone
+    grep -q '^RUN_FIRSTBOOT=true$' "$rootfs/$BT_INITHOOKS_DEFAULT"
+    grep -q '^SUDOADMIN=false$' "$rootfs/$BT_INITHOOKS_DEFAULT"
+}
+
+@test "mark_container: takes the first boot off tty1 with a systemd drop-in" {
+    rootfs=$(fake_rootfs)
+    run bt_mark_container "$rootfs"
+    [ "$status" -eq 0 ]
+    dropin="$rootfs/$BT_INITHOOKS_DROPIN"
+    [ -f "$dropin" ]
+    grep -q '^\[Service\]$' "$dropin"
+    grep -q '^StandardOutput=journal$' "$dropin"
+    grep -q '^StandardError=journal$' "$dropin"
+}
+
+@test "mark_container: a tree that already redirects is left redirecting" {
+    rootfs=$(fake_rootfs true)
+    run bt_mark_container "$rootfs"
+    [ "$status" -eq 0 ]
+    [ "$(grep -c '^REDIRECT_OUTPUT=true$' "$rootfs/$BT_INITHOOKS_DEFAULT")" -eq 1 ]
+}
+
+@test "mark_container: a rootfs with no inithooks defaults fails loudly" {
+    rootfs="$BATS_TEST_TMPDIR/bare"
+    mkdir -p "$rootfs"
+    run bt_mark_container "$rootfs"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"is not in the rootfs"* ]]
+}
+
+@test "mark_container: defaults that declare no REDIRECT_OUTPUT fail loudly" {
+    rootfs=$(fake_rootfs)
+    grep -v REDIRECT_OUTPUT "$rootfs/$BT_INITHOOKS_DEFAULT" > "$rootfs/trimmed"
+    mv "$rootfs/trimmed" "$rootfs/$BT_INITHOOKS_DEFAULT"
+    run bt_mark_container "$rootfs"
+    [ "$status" -eq 1 ]
+    [[ "$output" == *"declares no REDIRECT_OUTPUT"* ]]
+}
+
 @test "spec_targets: both paths the first boot reads, under the rootfs" {
     output=$(bt_spec_targets /r)
     [ "$output" = $'/r/etc/keel/instance.yaml\n/r/etc/inithooks.yaml' ]

tests/boot-test.sh

@@ -61,14 +61,16 @@ mkdir -p "$BT_ROOTFS"
 keel pull "$BT_APPLIANCE" --source "$BT_LAYERS_DIR" --cache-dir "$BT_CACHE_DIR" --non-interactive
 keel assemble "$BT_APPLIANCE" --rootfs "$BT_ROOTFS" --cache-dir "$BT_CACHE_DIR" --non-interactive
 
-# 2. The container marker, the instance description, the secrets it
-#    references and the conf the first boot hooks read. The marker under
-#    /var/lib/turnkey-info is what bt-container writes and what inspect
-#    reads to call the machine a container (managed_by: host); the conf is
-#    what makes the first boot headless, and without it 30rootpass and
-#    35pgsqlpass wait on a dialog forever.
+# 2. The container marks, the instance description, the secrets it
+#    references and the conf the first boot hooks read. bt_mark_container
+#    does what buildtasks' container patch does: the marker under
+#    /var/lib/turnkey-info that inspect reads to call the machine a
+#    container (managed_by: host), and REDIRECT_OUTPUT=true with a
+#    drop-in, without which a hook that prints a lot blocks writing to a
+#    tty1 nobody reads. The conf is what makes the first boot headless,
+#    and without it 30rootpass and 35pgsqlpass wait on a dialog forever.
 log "installing the description, the secrets and the conf into $BT_ROOTFS"
-install -D -m 0644 /dev/null "$BT_ROOTFS/var/lib/turnkey-info/inithooks.service/lxc"
+bt_mark_container "$BT_ROOTFS"
 install -d -m 0700 "$BT_ROOTFS/etc/keel/secrets"
 for target in $(bt_secret_targets "$BT_ROOTFS"); do
     bt_random_password > "$target"

tests/lib/boot-test-lib.sh

@@ -42,6 +42,13 @@ BT_DB_PROBE_ANSWER=1
 # the final name is a maintainer decision (brief section 11), so the test
 # installs the same file at both until then.
 BT_SPEC_PATHS="etc/keel/instance.yaml etc/inithooks.yaml"
+# What marks the tree as a container build, relative to the rootfs: the
+# marker file bt-container writes, the inithooks defaults whose
+# REDIRECT_OUTPUT it sets, and the drop-in that keeps the first boot off
+# tty1. See bt_mark_container.
+BT_CONTAINER_MARKER="var/lib/turnkey-info/inithooks.service/lxc"
+BT_INITHOOKS_DEFAULT="etc/default/inithooks"
+BT_INITHOOKS_DROPIN="etc/systemd/system/inithooks.service.d/container.conf"
 
 bt_usage() {
     cat <<USAGE
@@ -233,11 +240,26 @@ bt_lxc_config() {
     # bt_lxc_config NAME ROOTFS BRIDGE: an LXC config for a plain rootfs
     # directory on a bridge; the address comes from the bridge (SLAAC or
     # DHCPv6), the spec declares managed_by: host.
+    # The apparmor pair is not decoration. Under the stock container
+    # profile systemd cannot give a unit a mount namespace, so every unit
+    # with ProtectSystem or ProtectHome fails with status=226/NAMESPACE
+    # before its own first line runs. Measured in the container this test
+    # builds: systemd-journald, systemd-logind, systemd-sysusers,
+    # systemd-sysctl and tmp.mount all failed that way, which is why
+    # 15regen-sslcert and 95secupdates failed beside the database, and why
+    # the container had no journal to read when it was asked why. The
+    # cluster itself survived it, because postgresql@.service asks for no
+    # namespace; keel-mariadb's database did not. A generated profile with
+    # nesting allowed is what a container running systemd needs, and it is
+    # what the appliance containers on the build host have carried all
+    # along.
     cat <<CONFIG
 lxc.uts.name = $1
 lxc.rootfs.path = dir:$2
 lxc.include = /usr/share/lxc/config/common.conf
 lxc.arch = amd64
+lxc.apparmor.profile = generated
+lxc.apparmor.allow_nesting = 1
 lxc.net.0.type = veth
 lxc.net.0.link = $3
 lxc.net.0.name = eth0
@@ -246,6 +268,48 @@ lxc.start.auto = 0
 CONFIG
 }
 
+bt_mark_container() {
+    # bt_mark_container ROOTFS: make the tree look like the container build
+    # buildtasks produces, which is two things, both from its
+    # patches/container/conf:
+    #
+    #   the marker under /var/lib/turnkey-info, which inithooks' unit
+    #   conditions read and which `keel inspect` reads to call the machine a
+    #   container (network.managed_by: host), and
+    #
+    #   REDIRECT_OUTPUT=true in /etc/default/inithooks, which sends first
+    #   boot output to the log with a tail on the active console instead of
+    #   writing it straight to tty1,
+    #
+    # and a drop-in that keeps the first boot off tty1.
+    #
+    # The last two are not cosmetic. The layer ships the plain appliance
+    # inithooks.service, which runs the hooks with StandardOutput=tty on
+    # /dev/tty1; the unit a container image gets instead logs to syslog and
+    # the console. Nothing reads tty1 in a container nobody has attached to,
+    # so a hook that prints more than the terminal buffer holds blocks in
+    # the write and never returns. keel-nodebb found it the hard way, with
+    # `./nodebb setup` asleep in n_tty_write and a first boot that never
+    # finished; this is the same function, so the next hook that prints a
+    # lot does not find it again.
+    local rootfs=$1 defaults=$1/$BT_INITHOOKS_DEFAULT
+    install -D -m 0644 /dev/null "$rootfs/$BT_CONTAINER_MARKER" || return 1
+    if [ ! -f "$defaults" ]; then
+        echo "boot-test: $defaults is not in the rootfs" >&2
+        return 1
+    fi
+    sed -i '/REDIRECT_OUTPUT/ s/=.*/=true/' "$defaults" || return 1
+    if ! grep -q '^REDIRECT_OUTPUT=true$' "$defaults"; then
+        echo "boot-test: $defaults declares no REDIRECT_OUTPUT to set" >&2
+        return 1
+    fi
+    install -D -m 0644 /dev/stdin "$rootfs/$BT_INITHOOKS_DROPIN" <<DROPIN || return 1
+[Service]
+StandardOutput=journal
+StandardError=journal
+DROPIN
+}
+
 bt_spec_targets() {
     # bt_spec_targets ROOTFS: the paths the spec is installed at.
     local relative