diff --git a/COVERAGE.md b/COVERAGE.md index ce6a18c..409da5b 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -10,13 +10,13 @@ LXC as the acceptance test of a recipe, docs/org-plan.md section 1). | --- | --- | --- | --- | | overlay/usr/lib/inithooks/lib/postgresql.sh | tests/postgresql.bats (11 tests) | 100 percent (41/41) under kcov | every function and every branch | | overlay/usr/lib/inithooks/firstboot.d/36pgsqlverify | tests/hook.bats (10 tests) | 100 percent (15/15) under kcov | every path, including the two failures that matter | -| tests/lib/boot-test-lib.sh | tests/boot-test.bats (37 tests) | 100 percent (141/141) under kcov | argument parsing, address discovery, deadlines, the database, module, Webmin and diff verdicts | +| tests/lib/boot-test-lib.sh | tests/boot-test.bats (44 tests) | 100 percent (154/154) under kcov | argument parsing, address discovery, deadlines, the container marks, the database, module, Webmin and diff verdicts | | conf.d/main | the build | integration only | build time script, 0004 pragmatic limits | | tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root | | overlay ... firstboot.d/35pgsqlpass | common | not this repository | the hook that sets the password belongs to the `common` fork and is used, not rewritten | -Total over the three measured shell files: **100 percent (197/197)**, -58 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which +Total over the three measured shell files: **100 percent (210/210)**, +65 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which the workflow sets to 100, the measured number. It is only ever raised (decision 0006). @@ -24,7 +24,7 @@ the workflow sets to 100, the measured number. It is only ever raised kcov line coverage (threshold 100 percent): 100.00 41/41 postgresql.sh 100.00 15/15 36pgsqlverify - 100.00 141/141 boot-test-lib.sh + 100.00 154/154 boot-test-lib.sh ## What the hook tests cover @@ -52,9 +52,49 @@ the published layer from `https://mirror.keellinux.org/layers`, verifies it, assembles it, boots it in LXC and runs `tests/boot-test.sh`. Nothing is built there. -State on 2026-09-27: **the layer is not published yet, so the job skips -with a notice and passes.** It becomes a required status on `main` the day -the first `postgresql` layer reaches the mirror. +### What the gate found once the layer booted (2026-09-27) + +The `postgresql` layer was published (104,170,947 bytes, parent `core` +7acf2c53) and the job ran for real. It failed, on two defects at once, +one in the test and one in the layer: + + INFO: [35pgsqlpass] successfully completed + ERR: [36pgsqlverify] failed - exit code 1 + psql: error: connection to server at "::1", port 5432 failed: Connection refused + ERR: [15regen-sslcert] failed - exit code 1 + ERR: [95secupdates] failed - exit code 1 + +Reproduced on the build host from the published chain. + +1. **The test.** Under the stock LXC container apparmor profile systemd + cannot give a unit a mount namespace, so `systemd-journald`, + `systemd-logind`, `systemd-sysusers`, `systemd-sysctl` and + `tmp.mount` failed with `status=226/NAMESPACE`, which is why + `15regen-sslcert` and `95secupdates` failed and why the container had + no journal. `bt_lxc_config` now writes + `lxc.apparmor.profile = generated` and + `lxc.apparmor.allow_nesting = 1`, and `bt_mark_container` does what + buildtasks' container patch does, both ported from keel-nodebb. +2. **The layer**, and the hook was right to report it. The cluster was + online and listening on `127.0.0.1:5432` alone: Debian's `/etc/hosts` + maps `::1` to `ip6-localhost` and never to `localhost`, so + `listen_addresses = 'localhost'` binds the IPv4 loopback only. Fixed + by naming both addresses, in the changelog as version 2. + +Measured on the build host with both in place: every hook from +`01ipconfig` to `98finalize` completes, the only failed unit is +`inithooks-restart-getty1.service`, which needs a tty no container has, +and + + $ psql --username=postgres --host=::1 --port=5432 --no-password \ + --command='SELECT 1, current_user, inet_server_addr()' + 1|postgres|::1 + +with the declared password in `PGPASSWORD`, while the wrong password +gives `FATAL: password authentication failed for user "postgres"`. + +State on 2026-09-27: green once this and the layer rebuild land. It +becomes a required status on `main` then. ## Plan diff --git a/tests/README.md b/tests/README.md index ea76424..e1973f4 100644 --- a/tests/README.md +++ b/tests/README.md @@ -66,9 +66,16 @@ What it does, in order: 1. `keel pull` and `keel assemble` the chain (core, postgresql) into `//rootfs`. -2. Creates `var/lib/turnkey-info/inithooks.service/lxc` in the rootfs, the - marker `bt-container` writes and the one `keel inspect` reads to call - the machine a container (`network.managed_by: host`). +2. Marks the tree as a container build, which is what `bt_mark_container` + does and what buildtasks' `patches/container/conf` does for a real + container image: the marker + `var/lib/turnkey-info/inithooks.service/lxc` that `keel inspect` reads + to call the machine a container (`network.managed_by: host`), + `REDIRECT_OUTPUT=true` in `etc/default/inithooks`, and a drop-in giving + `inithooks.service` `StandardOutput=journal`. Without the last two the + hooks write to `/dev/tty1`, which nobody reads in a container, and the + first hook that prints more than the terminal buffer holds blocks there + forever. 3. Writes a random `root_password` and `db_password` under `etc/keel/secrets` (mode 0600) and installs `tests/instance.yaml` at `etc/keel/instance.yaml` and `etc/inithooks.yaml`. @@ -78,7 +85,11 @@ What it does, in order: would have nothing declared and no terminal, and `36pgsqlverify` would fail naming the field the description has to declare. 5. Writes an LXC config for that rootfs on the bridge and starts the - container. + container. The config asks for `lxc.apparmor.profile = generated` and + `lxc.apparmor.allow_nesting = 1`: without them systemd cannot give a + unit a mount namespace, so `systemd-journald`, `systemd-logind` and + `tmp.mount` fail with `status=226/NAMESPACE` and the hooks that need + them fail beside the database. 6. Waits for a global IPv6 address (`lxc-info -i`), then for the first boot to finish: `RUN_FIRSTBOOT=false` in the rootfs copy of `/etc/default/inithooks`, and then confconsole or an SSH banner. diff --git a/tests/boot-test.bats b/tests/boot-test.bats index 152b59e..13e5a36 100644 --- a/tests/boot-test.bats +++ b/tests/boot-test.bats @@ -242,6 +242,78 @@ never() { return 1; } [[ $output == *"lxc.net.0.type = veth"* ]] } +@test "lxc_config: the apparmor pair a container running systemd needs" { + output=$(bt_lxc_config keel-postgresql-boot-test /r/rootfs br0) + [[ $output == *"lxc.apparmor.profile = generated"* ]] + [[ $output == *"lxc.apparmor.allow_nesting = 1"* ]] +} + +fake_rootfs() { + # fake_rootfs [VALUE]: a scratch rootfs with an inithooks defaults file, + # REDIRECT_OUTPUT set to VALUE (default false), printed on stdout + local rootfs="$BATS_TEST_TMPDIR/rootfs-$RANDOM" + mkdir -p "$rootfs/etc/default" + cat > "$rootfs/$BT_INITHOOKS_DEFAULT" < "$rootfs/trimmed" + mv "$rootfs/trimmed" "$rootfs/$BT_INITHOOKS_DEFAULT" + run bt_mark_container "$rootfs" + [ "$status" -eq 1 ] + [[ "$output" == *"declares no REDIRECT_OUTPUT"* ]] +} + @test "spec_targets: both paths the first boot reads, under the rootfs" { output=$(bt_spec_targets /r) [ "$output" = $'/r/etc/keel/instance.yaml\n/r/etc/inithooks.yaml' ] diff --git a/tests/boot-test.sh b/tests/boot-test.sh index 47d54cf..fa70f81 100755 --- a/tests/boot-test.sh +++ b/tests/boot-test.sh @@ -61,14 +61,16 @@ mkdir -p "$BT_ROOTFS" keel pull "$BT_APPLIANCE" --source "$BT_LAYERS_DIR" --cache-dir "$BT_CACHE_DIR" --non-interactive keel assemble "$BT_APPLIANCE" --rootfs "$BT_ROOTFS" --cache-dir "$BT_CACHE_DIR" --non-interactive -# 2. The container marker, the instance description, the secrets it -# references and the conf the first boot hooks read. The marker under -# /var/lib/turnkey-info is what bt-container writes and what inspect -# reads to call the machine a container (managed_by: host); the conf is -# what makes the first boot headless, and without it 30rootpass and -# 35pgsqlpass wait on a dialog forever. +# 2. The container marks, the instance description, the secrets it +# references and the conf the first boot hooks read. bt_mark_container +# does what buildtasks' container patch does: the marker under +# /var/lib/turnkey-info that inspect reads to call the machine a +# container (managed_by: host), and REDIRECT_OUTPUT=true with a +# drop-in, without which a hook that prints a lot blocks writing to a +# tty1 nobody reads. The conf is what makes the first boot headless, +# and without it 30rootpass and 35pgsqlpass wait on a dialog forever. log "installing the description, the secrets and the conf into $BT_ROOTFS" -install -D -m 0644 /dev/null "$BT_ROOTFS/var/lib/turnkey-info/inithooks.service/lxc" +bt_mark_container "$BT_ROOTFS" install -d -m 0700 "$BT_ROOTFS/etc/keel/secrets" for target in $(bt_secret_targets "$BT_ROOTFS"); do bt_random_password > "$target" diff --git a/tests/lib/boot-test-lib.sh b/tests/lib/boot-test-lib.sh index 8903fbe..0da5517 100644 --- a/tests/lib/boot-test-lib.sh +++ b/tests/lib/boot-test-lib.sh @@ -42,6 +42,13 @@ BT_DB_PROBE_ANSWER=1 # the final name is a maintainer decision (brief section 11), so the test # installs the same file at both until then. BT_SPEC_PATHS="etc/keel/instance.yaml etc/inithooks.yaml" +# What marks the tree as a container build, relative to the rootfs: the +# marker file bt-container writes, the inithooks defaults whose +# REDIRECT_OUTPUT it sets, and the drop-in that keeps the first boot off +# tty1. See bt_mark_container. +BT_CONTAINER_MARKER="var/lib/turnkey-info/inithooks.service/lxc" +BT_INITHOOKS_DEFAULT="etc/default/inithooks" +BT_INITHOOKS_DROPIN="etc/systemd/system/inithooks.service.d/container.conf" bt_usage() { cat <&2 + return 1 + fi + sed -i '/REDIRECT_OUTPUT/ s/=.*/=true/' "$defaults" || return 1 + if ! grep -q '^REDIRECT_OUTPUT=true$' "$defaults"; then + echo "boot-test: $defaults declares no REDIRECT_OUTPUT to set" >&2 + return 1 + fi + install -D -m 0644 /dev/stdin "$rootfs/$BT_INITHOOKS_DROPIN" <