From 95ba3d6f1ad4222b7e4296093ea2c1a1f0755321 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marcos=20M=C3=A9ndez?= Date: Sun, 27 Sep 2026 06:53:22 +0000 Subject: [PATCH] Give the boot test container a systemd the first boot can rely on The other half of what the gate found once the layer booted. The layer's own half, the cluster listening on the IPv4 loopback alone, is fixed in the pull request before this one; this is the container the test builds. Under the stock LXC container apparmor profile systemd cannot give a unit a mount namespace, so every unit that asks for one fails with status=226/NAMESPACE before its own first line runs. Measured on the build host in a container booted exactly as this test boots it, systemd-journald, systemd-logind, systemd-sysusers, systemd-sysctl and tmp.mount had all failed that way. That is why 15regen-sslcert and 95secupdates failed beside the database, and why the container had no journal to read when it was asked why. The cluster itself survived it, because postgresql@.service asks for no namespace, which is exactly what made the report confusing: the database was up and unreachable for an unrelated reason. Two things fix it, both ported from keel-nodebb rather than invented again, since that repository met both first and keel-mariadb is taking the same two: bt_lxc_config now writes lxc.apparmor.profile = generated and lxc.apparmor.allow_nesting = 1, which is what the appliance containers on the build host have carried all along (keel-nodebb pull request 8). bt_mark_container replaces the single marker line and does all three things a container build does: the marker under /var/lib/turnkey-info, REDIRECT_OUTPUT=true in etc/default/inithooks, and a drop-in giving inithooks.service StandardOutput=journal. The layer ships the plain appliance unit, which runs the hooks on /dev/tty1; nothing reads tty1 in a container nobody has attached to, so a hook that prints past the terminal buffer blocks there forever. It hung keel-nodebb's first boot (pull request 9). No hook here prints that much today, which is exactly why it should not be left for the next one to find. Measured on the build host against the published chain with both in place, and with the layer's listen_addresses corrected: every hook from 01ipconfig to 98finalize completes, including 15regen-sslcert and 95secupdates, and the only failed unit is inithooks-restart-getty1.service, which wants a tty no container has. Seven tests for the two changes; boot-test-lib.sh stays at 100 percent (154/154) and the three measured files total 100 (210/210) over 65 bats tests. Nothing in the layer changes here, so no changelog entry. --- COVERAGE.md | 54 ++++++++++++++++++++++++---- tests/README.md | 19 +++++++--- tests/boot-test.bats | 72 ++++++++++++++++++++++++++++++++++++++ tests/boot-test.sh | 16 +++++---- tests/lib/boot-test-lib.sh | 64 +++++++++++++++++++++++++++++++++ 5 files changed, 207 insertions(+), 18 deletions(-) diff --git a/COVERAGE.md b/COVERAGE.md index ce6a18c..409da5b 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -10,13 +10,13 @@ LXC as the acceptance test of a recipe, docs/org-plan.md section 1). | --- | --- | --- | --- | | overlay/usr/lib/inithooks/lib/postgresql.sh | tests/postgresql.bats (11 tests) | 100 percent (41/41) under kcov | every function and every branch | | overlay/usr/lib/inithooks/firstboot.d/36pgsqlverify | tests/hook.bats (10 tests) | 100 percent (15/15) under kcov | every path, including the two failures that matter | -| tests/lib/boot-test-lib.sh | tests/boot-test.bats (37 tests) | 100 percent (141/141) under kcov | argument parsing, address discovery, deadlines, the database, module, Webmin and diff verdicts | +| tests/lib/boot-test-lib.sh | tests/boot-test.bats (44 tests) | 100 percent (154/154) under kcov | argument parsing, address discovery, deadlines, the container marks, the database, module, Webmin and diff verdicts | | conf.d/main | the build | integration only | build time script, 0004 pragmatic limits | | tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root | | overlay ... firstboot.d/35pgsqlpass | common | not this repository | the hook that sets the password belongs to the `common` fork and is used, not rewritten | -Total over the three measured shell files: **100 percent (197/197)**, -58 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which +Total over the three measured shell files: **100 percent (210/210)**, +65 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which the workflow sets to 100, the measured number. It is only ever raised (decision 0006). @@ -24,7 +24,7 @@ the workflow sets to 100, the measured number. It is only ever raised kcov line coverage (threshold 100 percent): 100.00 41/41 postgresql.sh 100.00 15/15 36pgsqlverify - 100.00 141/141 boot-test-lib.sh + 100.00 154/154 boot-test-lib.sh ## What the hook tests cover @@ -52,9 +52,49 @@ the published layer from `https://mirror.keellinux.org/layers`, verifies it, assembles it, boots it in LXC and runs `tests/boot-test.sh`. Nothing is built there. -State on 2026-09-27: **the layer is not published yet, so the job skips -with a notice and passes.** It becomes a required status on `main` the day -the first `postgresql` layer reaches the mirror. +### What the gate found once the layer booted (2026-09-27) + +The `postgresql` layer was published (104,170,947 bytes, parent `core` +7acf2c53) and the job ran for real. It failed, on two defects at once, +one in the test and one in the layer: + + INFO: [35pgsqlpass] successfully completed + ERR: [36pgsqlverify] failed - exit code 1 + psql: error: connection to server at "::1", port 5432 failed: Connection refused + ERR: [15regen-sslcert] failed - exit code 1 + ERR: [95secupdates] failed - exit code 1 + +Reproduced on the build host from the published chain. + +1. **The test.** Under the stock LXC container apparmor profile systemd + cannot give a unit a mount namespace, so `systemd-journald`, + `systemd-logind`, `systemd-sysusers`, `systemd-sysctl` and + `tmp.mount` failed with `status=226/NAMESPACE`, which is why + `15regen-sslcert` and `95secupdates` failed and why the container had + no journal. `bt_lxc_config` now writes + `lxc.apparmor.profile = generated` and + `lxc.apparmor.allow_nesting = 1`, and `bt_mark_container` does what + buildtasks' container patch does, both ported from keel-nodebb. +2. **The layer**, and the hook was right to report it. The cluster was + online and listening on `127.0.0.1:5432` alone: Debian's `/etc/hosts` + maps `::1` to `ip6-localhost` and never to `localhost`, so + `listen_addresses = 'localhost'` binds the IPv4 loopback only. Fixed + by naming both addresses, in the changelog as version 2. + +Measured on the build host with both in place: every hook from +`01ipconfig` to `98finalize` completes, the only failed unit is +`inithooks-restart-getty1.service`, which needs a tty no container has, +and + + $ psql --username=postgres --host=::1 --port=5432 --no-password \ + --command='SELECT 1, current_user, inet_server_addr()' + 1|postgres|::1 + +with the declared password in `PGPASSWORD`, while the wrong password +gives `FATAL: password authentication failed for user "postgres"`. + +State on 2026-09-27: green once this and the layer rebuild land. It +becomes a required status on `main` then. ## Plan diff --git a/tests/README.md b/tests/README.md index ea76424..e1973f4 100644 --- a/tests/README.md +++ b/tests/README.md @@ -66,9 +66,16 @@ What it does, in order: 1. `keel pull` and `keel assemble` the chain (core, postgresql) into `//rootfs`. -2. Creates `var/lib/turnkey-info/inithooks.service/lxc` in the rootfs, the - marker `bt-container` writes and the one `keel inspect` reads to call - the machine a container (`network.managed_by: host`). +2. Marks the tree as a container build, which is what `bt_mark_container` + does and what buildtasks' `patches/container/conf` does for a real + container image: the marker + `var/lib/turnkey-info/inithooks.service/lxc` that `keel inspect` reads + to call the machine a container (`network.managed_by: host`), + `REDIRECT_OUTPUT=true` in `etc/default/inithooks`, and a drop-in giving + `inithooks.service` `StandardOutput=journal`. Without the last two the + hooks write to `/dev/tty1`, which nobody reads in a container, and the + first hook that prints more than the terminal buffer holds blocks there + forever. 3. Writes a random `root_password` and `db_password` under `etc/keel/secrets` (mode 0600) and installs `tests/instance.yaml` at `etc/keel/instance.yaml` and `etc/inithooks.yaml`. @@ -78,7 +85,11 @@ What it does, in order: would have nothing declared and no terminal, and `36pgsqlverify` would fail naming the field the description has to declare. 5. Writes an LXC config for that rootfs on the bridge and starts the - container. + container. The config asks for `lxc.apparmor.profile = generated` and + `lxc.apparmor.allow_nesting = 1`: without them systemd cannot give a + unit a mount namespace, so `systemd-journald`, `systemd-logind` and + `tmp.mount` fail with `status=226/NAMESPACE` and the hooks that need + them fail beside the database. 6. Waits for a global IPv6 address (`lxc-info -i`), then for the first boot to finish: `RUN_FIRSTBOOT=false` in the rootfs copy of `/etc/default/inithooks`, and then confconsole or an SSH banner. diff --git a/tests/boot-test.bats b/tests/boot-test.bats index 152b59e..13e5a36 100644 --- a/tests/boot-test.bats +++ b/tests/boot-test.bats @@ -242,6 +242,78 @@ never() { return 1; } [[ $output == *"lxc.net.0.type = veth"* ]] } +@test "lxc_config: the apparmor pair a container running systemd needs" { + output=$(bt_lxc_config keel-postgresql-boot-test /r/rootfs br0) + [[ $output == *"lxc.apparmor.profile = generated"* ]] + [[ $output == *"lxc.apparmor.allow_nesting = 1"* ]] +} + +fake_rootfs() { + # fake_rootfs [VALUE]: a scratch rootfs with an inithooks defaults file, + # REDIRECT_OUTPUT set to VALUE (default false), printed on stdout + local rootfs="$BATS_TEST_TMPDIR/rootfs-$RANDOM" + mkdir -p "$rootfs/etc/default" + cat > "$rootfs/$BT_INITHOOKS_DEFAULT" < "$rootfs/trimmed" + mv "$rootfs/trimmed" "$rootfs/$BT_INITHOOKS_DEFAULT" + run bt_mark_container "$rootfs" + [ "$status" -eq 1 ] + [[ "$output" == *"declares no REDIRECT_OUTPUT"* ]] +} + @test "spec_targets: both paths the first boot reads, under the rootfs" { output=$(bt_spec_targets /r) [ "$output" = $'/r/etc/keel/instance.yaml\n/r/etc/inithooks.yaml' ] diff --git a/tests/boot-test.sh b/tests/boot-test.sh index 47d54cf..fa70f81 100755 --- a/tests/boot-test.sh +++ b/tests/boot-test.sh @@ -61,14 +61,16 @@ mkdir -p "$BT_ROOTFS" keel pull "$BT_APPLIANCE" --source "$BT_LAYERS_DIR" --cache-dir "$BT_CACHE_DIR" --non-interactive keel assemble "$BT_APPLIANCE" --rootfs "$BT_ROOTFS" --cache-dir "$BT_CACHE_DIR" --non-interactive -# 2. The container marker, the instance description, the secrets it -# references and the conf the first boot hooks read. The marker under -# /var/lib/turnkey-info is what bt-container writes and what inspect -# reads to call the machine a container (managed_by: host); the conf is -# what makes the first boot headless, and without it 30rootpass and -# 35pgsqlpass wait on a dialog forever. +# 2. The container marks, the instance description, the secrets it +# references and the conf the first boot hooks read. bt_mark_container +# does what buildtasks' container patch does: the marker under +# /var/lib/turnkey-info that inspect reads to call the machine a +# container (managed_by: host), and REDIRECT_OUTPUT=true with a +# drop-in, without which a hook that prints a lot blocks writing to a +# tty1 nobody reads. The conf is what makes the first boot headless, +# and without it 30rootpass and 35pgsqlpass wait on a dialog forever. log "installing the description, the secrets and the conf into $BT_ROOTFS" -install -D -m 0644 /dev/null "$BT_ROOTFS/var/lib/turnkey-info/inithooks.service/lxc" +bt_mark_container "$BT_ROOTFS" install -d -m 0700 "$BT_ROOTFS/etc/keel/secrets" for target in $(bt_secret_targets "$BT_ROOTFS"); do bt_random_password > "$target" diff --git a/tests/lib/boot-test-lib.sh b/tests/lib/boot-test-lib.sh index 8903fbe..0da5517 100644 --- a/tests/lib/boot-test-lib.sh +++ b/tests/lib/boot-test-lib.sh @@ -42,6 +42,13 @@ BT_DB_PROBE_ANSWER=1 # the final name is a maintainer decision (brief section 11), so the test # installs the same file at both until then. BT_SPEC_PATHS="etc/keel/instance.yaml etc/inithooks.yaml" +# What marks the tree as a container build, relative to the rootfs: the +# marker file bt-container writes, the inithooks defaults whose +# REDIRECT_OUTPUT it sets, and the drop-in that keeps the first boot off +# tty1. See bt_mark_container. +BT_CONTAINER_MARKER="var/lib/turnkey-info/inithooks.service/lxc" +BT_INITHOOKS_DEFAULT="etc/default/inithooks" +BT_INITHOOKS_DROPIN="etc/systemd/system/inithooks.service.d/container.conf" bt_usage() { cat <&2 + return 1 + fi + sed -i '/REDIRECT_OUTPUT/ s/=.*/=true/' "$defaults" || return 1 + if ! grep -q '^REDIRECT_OUTPUT=true$' "$defaults"; then + echo "boot-test: $defaults declares no REDIRECT_OUTPUT to set" >&2 + return 1 + fi + install -D -m 0644 /dev/stdin "$rootfs/$BT_INITHOOKS_DROPIN" <