A boot test that proves the declared password, and the coverage gate

.github/workflows/tests.yml

@@ -19,16 +19,16 @@ permissions:
 
 jobs:
   tests:
-    # Threshold 0 is the bootstrap case of test-shell.yml: there is no
-    # tests/coverage.sh on this branch yet, so the job passes with a notice
-    # and the check exists, which is what lets main be protected from the
-    # first day. It is raised to the measured number when the tests land,
-    # and only ever raised (decision 0006). This job produces the check
-    # "tests / coverage", the required status on main. The job id is part
-    # of that name, so renaming it silently detaches the protection rule.
+    # tests/coverage.sh measures three shell files (COVERAGE.md): the
+    # first boot library, the verification hook this layer adds and the
+    # boot test's library, all three at 100 percent. The threshold is the
+    # measured number and is only ever raised (decision 0006). This job
+    # produces the check "tests / coverage", the required status on main.
+    # The job id is part of that name, so renaming it silently detaches
+    # the protection rule.
     uses: keel-linux/.github/.github/workflows/test-shell.yml@main
     with:
-      threshold: 0
+      threshold: 100
   appliance:
     # The boot test (org-plan section 1): the postgresql layer is fetched from
     # https://mirror.keellinux.org/layers, verified, assembled into a

COVERAGE.md

@@ -0,0 +1,67 @@
+# Coverage
+
+Standard: decisions 0003 (90 percent per repository, 95 for code the
+project writes) and 0004 (bats plus kcov for shell; a build and a boot on
+LXC as the acceptance test of a recipe, docs/org-plan.md section 1).
+
+## Measured 2026-09-27
+
+| File | Test | Lines | Note |
+| --- | --- | --- | --- |
+| overlay/usr/lib/inithooks/lib/postgresql.sh | tests/postgresql.bats (11 tests) | 100 percent (41/41) under kcov | every function and every branch |
+| overlay/usr/lib/inithooks/firstboot.d/36pgsqlverify | tests/hook.bats (10 tests) | 100 percent (15/15) under kcov | every path, including the two failures that matter |
+| tests/lib/boot-test-lib.sh | tests/boot-test.bats (37 tests) | 100 percent (141/141) under kcov | argument parsing, address discovery, deadlines, the database, module, Webmin and diff verdicts |
+| conf.d/main | the build | integration only | build time script, 0004 pragmatic limits |
+| tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root |
+| overlay ... firstboot.d/35pgsqlpass | common | not this repository | the hook that sets the password belongs to the `common` fork and is used, not rewritten |
+
+Total over the three measured shell files: **100 percent (197/197)**,
+58 bats tests. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which
+the workflow sets to 100, the measured number. It is only ever raised
+(decision 0006).
+
+    $ COVERAGE_THRESHOLD=100 tests/coverage.sh
+    kcov line coverage (threshold 100 percent):
+     100.00  41/41  postgresql.sh
+     100.00  15/15  36pgsqlverify
+     100.00  141/141  boot-test-lib.sh
+
+## What the hook tests cover
+
+The hook is executed for real against scratch directories, with PATH stubs
+for `systemctl`, `pg_isready` and `psql` under a scratch `INITHOOKS_PATH`
+whose `lib` is a symlink to the real library, so kcov measures the file the
+layer ships. The `psql` stub answers the probe only for the password the
+description declared, the way a server does. No test needs root, a
+database or a network.
+
+What they are about is the half of the password defect that is this
+layer's: the declared password is proved against the database, a password
+the database refuses fails the hook, an answer that is not the probe's
+fails it, nothing declared fails naming the field and never prompts, a
+`PGSQL_PASS` in the conf is not a password and is ignored, `APP_DB_USER`
+renames the role and a name that would need quoting is refused before
+anything runs, and the cluster is started and waited for on the address the
+client will use.
+
+## The appliance gate
+
+`appliance / build-and-boot` runs through the organization's
+`test-appliance.yml` on the self-hosted `keel-lxc` runner, which fetches
+the published layer from `https://mirror.keellinux.org/layers`, verifies
+it, assembles it, boots it in LXC and runs `tests/boot-test.sh`. Nothing is
+built there.
+
+State on 2026-09-27: **the layer is not published yet, so the job skips
+with a notice and passes.** It becomes a required status on `main` the day
+the first `postgresql` layer reaches the mirror.
+
+## Plan
+
+- Publish the layer, then require `appliance / build-and-boot` on `main`.
+- Measure `conf.d/main`. A build time script that runs inside a chroot as
+  root is the case decision 0003 splits, and what is left here after the
+  logic moved to `lib/postgresql.sh` is SQL, three assertions about the
+  cluster configuration and the apt calls.
+- `35pgsqlpass` and `bin/pgsqlconf.py` belong to the `common` fork and are
+  measured there; `common`'s own COVERAGE.md carries that plan.

tests/README.md

@@ -0,0 +1,97 @@
+# Tests
+
+What a test means for a layer recipe is written in `COVERAGE.md`: the
+recipe builds, the result boots in an LXC container, its first boot
+completes headless from an instance description, the database accepts the
+password that description declared, the panel has the module for it, and
+the machine matches the description.
+
+## Layout
+
+- `boot-test.sh`: the boot test. `test-appliance.yml` (reusable workflow of
+  `keel-linux/.github`) runs it on the self-hosted LXC runner after pulling
+  the layers from `https://mirror.keellinux.org/layers` and checking them
+  with `keel verify`. It is the thin main: assemble, mark the tree as a
+  container, install the description, the secrets and the conf, start the
+  container, wait, connect to the database, check Webmin, `keel diff`. It
+  builds nothing, so it needs no fab, deck or buildtasks.
+- `lib/boot-test-lib.sh`: the logic (argument parsing, address discovery
+  from `lxc-info`, waiting with a deadline, the secret files, the client
+  call, the database, module and Webmin verdicts, the diff verdict), as
+  functions with no side effects, per decision 0004. Same shape as the one
+  in keel-core, keel-nodebb and keel-mariadb.
+- `boot-test.bats`: unit tests of that library. `lxc-info` is a stub first
+  in `PATH`; the clock and `sleep` are functions. No root, no network, no
+  LXC, no database.
+- `postgresql.bats`: unit tests of
+  `overlay/usr/lib/inithooks/lib/postgresql.sh`, the logic behind the
+  first boot hook `36pgsqlverify`.
+- `hook.bats`: `36pgsqlverify` itself, run for real against scratch
+  directories with every system command stubbed. The hook that sets the
+  password, `35pgsqlpass`, belongs to the `common` fork and is tested
+  there; this layer uses it rather than rewriting it.
+- `coverage.sh`: runs the bats suite under kcov and fails when any measured
+  file is below `COVERAGE_THRESHOLD` (default 100).
+- `instance.yaml`: the description the test container boots from. It
+  declares `secrets.db_password` from a file, which is the point of the
+  test.
+
+## Unit tests and coverage
+
+Debian packages `bats` (1.11) and `kcov` (43); no root:
+
+    bats tests/postgresql.bats
+    bats tests/hook.bats
+    bats tests/boot-test.bats
+    COVERAGE_THRESHOLD=100 tests/coverage.sh
+
+`COVERAGE_DIR=coverage tests/coverage.sh` keeps the kcov reports.
+
+## The boot test by hand
+
+Needs root, `keel` on `PATH`, LXC (`lxc-start`, `lxc-info`, `lxc-attach`,
+`lxc-stop`), `curl`, and a bridge with IPv6 router advertisements or
+DHCPv6.
+
+    tests/boot-test.sh postgresql --layers-dir https://mirror.keellinux.org/layers \
+        --bridge lxcbr0
+
+`--layers-dir` is a directory or an http(s) URL, so on the build host it is
+`/mnt/builds/layers` and on a runner it is the mirror. The other useful
+options are `--bridge`, `--cache-dir`, `--lxc-path`, `--name`, `--timeout`
+and `--keep` (leaves the container running; then `lxc-attach -n <name>`).
+`tests/boot-test.sh --help` lists them all.
+
+What it does, in order:
+
+1. `keel pull` and `keel assemble` the chain (core, postgresql) into
+   `<lxc-path>/<name>/rootfs`.
+2. Creates `var/lib/turnkey-info/inithooks.service/lxc` in the rootfs, the
+   marker `bt-container` writes and the one `keel inspect` reads to call
+   the machine a container (`network.managed_by: host`).
+3. Writes a random `root_password` and `db_password` under
+   `etc/keel/secrets` (mode 0600) and installs `tests/instance.yaml` at
+   `etc/keel/instance.yaml` and `etc/inithooks.yaml`.
+4. Renders the description into the rootfs `etc/inithooks.conf` with
+   `keel spec apply`, from a copy whose secret references point inside the
+   rootfs. Without the conf the first boot is not headless: `30rootpass`
+   would have nothing declared and no terminal, and `36pgsqlverify`
+   would fail naming the field the description has to declare.
+5. Writes an LXC config for that rootfs on the bridge and starts the
+   container.
+6. Waits for a global IPv6 address (`lxc-info -i`), then for the first boot
+   to finish: `RUN_FIRSTBOOT=false` in the rootfs copy of
+   `/etc/default/inithooks`, and then confconsole or an SSH banner.
+7. **Connects to the database.** `psql --username=postgres --host=::1
+   --port=5432 --no-password --command='SELECT 1'` inside the container,
+   with the declared password in `PGPASSWORD`. `--no-password` makes psql
+   fail rather than prompt, so a password that did not arrive is an error
+   and not a hung test. The password comes from the secret
+   file the test wrote in step 3 and from nowhere else, so a row back is
+   the proof that the declarative path carried it end to end. A listening
+   port would prove nothing: the database listens whatever password it
+   ended up with.
+8. Checks `webmin-postgresql` is installed and that Webmin answers over IPv6 on
+   12321.
+9. Runs `keel diff --root <rootfs> --spec tests/instance.yaml`; exit 0 or
+   13 (no drift) passes.