Make a build that installs yesterday's packages fail

COVERAGE.md

@@ -12,18 +12,20 @@ acceptance test of an appliance recipe, docs/org-plan.md section 1).
 | overlay/usr/lib/inithooks/firstboot.d/40nodebb | tests/hook.bats (15 tests) | 96.97 percent (32/33) under kcov | the one uncovered line is inside the dialog loop, which needs a terminal |
 | overlay/etc/nginx/* | tests/nginx.bats (12 tests) | not executable | asserted as content: the geo variable, the map, the listeners, the proxy headers |
 | tests/lib/boot-test-lib.sh | tests/boot-test.bats (34 tests) | 100 percent (124/124) under kcov | the logic of the boot test: argument parsing, address discovery, deadlines, the HTTP and diff verdicts |
+| bin/keel-archive-check | tests/archive-check.bats (8 tests) | 100 percent (26/26) under kcov | the build time check that the archive copy in the build tree is the live archive |
+| conf.d/zz-project-packages | tests/project-packages.bats (13 tests) | 100 percent (29/29) under kcov | the build time check that each project package is the candidate of the archive, and a project build |
 | overlay/usr/lib/inithooks/bin/nodebb.py | none | 0 | dialog wrapper, only reached with a terminal attached |
 | conf.d/main | tests/boot-test.sh (build step) | integration only | build time script, 0004 pragmatic limits |
 | tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root |
 
-Total over the three measured shell files: 99.00 percent (199/201) before the
-terminal test, 99.50 percent (200/201) with it.
+Total over the five measured shell files: 99.22 percent (254/256) before the
+terminal test, 99.61 percent (255/256) with it.
 
 `tests/coverage.sh` runs the whole bats suite under kcov, measures the
-library, the first boot hook and the boot test's own library, and fails below
-95 percent; the workflow runs it on every pull request through the
-organization's `test-shell.yml` and produces the required check
-`tests / coverage` on `main`.
+library, the first boot hook, the boot test's own library and the two build
+time checks, and fails below 95 percent; the workflow runs it on every pull
+request through the organization's `test-shell.yml` and produces the required
+check `tests / coverage` on `main`.
 
 The appliance test runs through the organization's `test-appliance.yml` on the
 self-hosted `keel-lxc` runner, which fetches the published layer from
@@ -58,6 +60,23 @@ check, the service calls, the asset build switch, both idempotence paths, the
 refusal of an address nginx would reject, the empty trusted proxy, the domain
 fallback, the missing password with and without a terminal.
 
+### What the build time checks cover
+
+The two build time scripts also run for real, against scratch trees, with
+`dpkg`, `dpkg-query` and `apt-cache` as PATH stubs driven by fixture files, so
+no test needs a chroot, apt or root. `bin/keel-archive-check`: a faithful
+copy, a copy of an older archive (the message names the version that changed),
+a missing copy, a missing source index and the argument errors.
+`conf.d/zz-project-packages`: the three packages at the versions the archive
+offers, the same three at versions nobody has published yet (the script names
+no version, so that passes unchanged), yesterday's package against today's
+archive, a candidate that is not what the archive offers, an upstream build of
+the same version, a package the archive does not offer, an archive that offers
+two of them, a package that is not installed, a half configured one, a build
+with no project archive in its source list, a distribution the archive has not
+got, and the removal of the build time source with the disabled
+`apt.keellinux.org` entry left in place.
+
 ## Plan
 
 - Rebuild and publish `nodebb` now that pull request 1 fixed the conffile

Makefile

@@ -11,16 +11,34 @@ include $(FAB_PATH)/common/mk/turnkey.mk
 # The project's own packages (inithooks, confconsole, keel) come from the
 # build host's APT repository during the build only. The repository is copied
 # into the bootstrap and listed as a [trusted=yes] file source, because the
-# staging distribution is unsigned; conf.d/main removes both from the image
-# and leaves the future apt.keellinux.org entry in place, disabled.
+# staging distribution is unsigned; conf.d/zz-project-packages checks what was
+# installed against that copy, removes both from the image and leaves the
+# future apt.keellinux.org entry in place, disabled.
 KEEL_APT_REPO ?= /srv/keel-apt/repo
 KEEL_APT_DIST ?= trixie-staging
+KEEL_ARCHIVE_CHECK = $(CURDIR)/bin/keel-archive-check $(KEEL_APT_REPO)
 
+# The copy is made fresh and then proved: bin/keel-archive-check compares the
+# copied package index with the live one and stops the build when they differ.
 define _keel_bootstrap/post
-	
+
 	mkdir -p $O/bootstrap/srv/keel-apt/repo;
+	rm -rf $O/bootstrap/srv/keel-apt/repo/dists $O/bootstrap/srv/keel-apt/repo/pool;
 	cp -a $(KEEL_APT_REPO)/dists $(KEEL_APT_REPO)/pool $O/bootstrap/srv/keel-apt/repo/;
+	$(KEEL_ARCHIVE_CHECK) $O/bootstrap $(KEEL_APT_DIST) $(FAB_ARCH) bootstrap;
 	echo "deb [trusted=yes] file:///srv/keel-apt/repo $(KEEL_APT_DIST) main" > $O/bootstrap/etc/apt/sources.list.d/keel-staging.list;
 	fab-chroot $O/bootstrap "apt-get update";
 endef
 bootstrap/post += $(_keel_bootstrap/post)
+
+# bootstrap is a stamped target, so a second build of the same product reuses
+# the copy the first one made and a check in bootstrap/post does not run at
+# all. On 2026-09-26 "make clean" failed on a busy deck, the stamps survived,
+# and the rebuild installed the packages the archive had held that morning
+# without a word. So the tree that is about to be configured is checked on
+# every build, whether or not this build made the bootstrap.
+define _keel_root.patched/pre
+
+	$(KEEL_ARCHIVE_CHECK) $O/root.patched $(KEEL_APT_DIST) $(FAB_ARCH) root.patched;
+endef
+root.patched/pre += $(_keel_root.patched/pre)

README.rst

@@ -58,9 +58,24 @@ appliance.
 The project's own packages (inithooks, confconsole, keel) are listed in the
 plan and resolved, during the build only, from the build host's repository
 copied into the bootstrap as a ``[trusted=yes] file:///srv/keel-apt/repo``
-source (``Makefile``, ``bootstrap/post``). ``conf.d/main`` removes that source
-from the image and leaves ``/etc/apt/sources.list.d/keel.sources`` pointing at
-the future ``apt.keellinux.org``, disabled, with the origin pin in
+source (``Makefile``, ``bootstrap/post``).
+
+That copy has to be the archive as it is at build time: fab stamps the
+bootstrap target, so a rebuild would otherwise reuse the copy an earlier build
+made and install packages the archive no longer offers.
+``bin/keel-archive-check`` compares the copied package index with the live one
+where the copy is made, and again on the tree that is about to be configured,
+and stops the build when they differ.
+
+``conf.d/zz-project-packages`` runs last. For each project package it checks
+that the archive offers exactly one version, that apt's candidate is that
+version, that the version comes from the project archive rather than an
+upstream source, and that the installed package is that candidate and
+configured. No version is written down anywhere, so a rebuild made after a
+publication either carries the new versions or fails. It then removes the
+build time source from the image and leaves
+``/etc/apt/sources.list.d/keel.sources`` pointing at the future
+``apt.keellinux.org``, disabled, with the origin pin in
 ``/etc/apt/preferences.d/keel``.
 
 First boot

bin/keel-archive-check

@@ -0,0 +1,60 @@
+#!/bin/bash
+# The build reads the project's APT archive from a copy inside the build tree.
+# This checks that the copy is the archive as it is right now, byte for byte,
+# and stops the build when it is not.
+#
+# Why it exists: fab stamps the bootstrap target, so a second build of the same
+# product reuses the bootstrap the first one made, copy of the archive and all.
+# On 2026-09-26 "make clean" failed on a busy deck, the stamps survived, and
+# the rebuild installed the packages the archive had held that morning. The
+# build said nothing, because every step of it succeeded.
+#
+#   keel-archive-check SOURCE_REPO TREE DIST ARCH [LABEL]
+#
+# SOURCE_REPO  the live archive on the build host, e.g. /srv/keel-apt/repo
+# TREE         the build tree that carries the copy under srv/keel-apt/repo
+# DIST         the distribution, e.g. trixie-staging
+# ARCH         the Debian architecture, e.g. amd64
+# LABEL        name of the build step, for the messages (default: TREE)
+#
+# Exit 0 when the copy matches, 1 otherwise.
+set -eu
+
+usage() {
+    echo "usage: $(basename "$0") SOURCE_REPO TREE DIST ARCH [LABEL]" >&2
+    exit 1
+}
+
+[ $# -ge 4 ] && [ $# -le 5 ] || usage
+source_repo=$1
+tree=$2
+dist=$3
+arch=$4
+label=${5:-$2}
+
+for value in "$source_repo" "$tree" "$dist" "$arch"; do
+    [ -n "$value" ] || usage
+done
+
+index=dists/$dist/main/binary-$arch/Packages
+source_index=$source_repo/$index
+copy_index=$tree/srv/keel-apt/repo/$index
+
+fatal() {
+    echo "FATAL [archive-check $label]: $*" >&2
+    exit 1
+}
+
+[ -f "$source_index" ] || fatal "the archive has no index at $source_index"
+[ -f "$copy_index" ] || fatal "the build tree has no archive index at $copy_index"
+
+if ! cmp -s "$source_index" "$copy_index"; then
+    echo "FATAL [archive-check $label]: $copy_index is not $source_index" >&2
+    diff -u "$copy_index" "$source_index" >&2 || true
+    echo "The build would install the packages the archive offered when this" >&2
+    echo "bootstrap was made, not the ones it offers now. Remove the build" >&2
+    echo "directory (or at least build/stamps/bootstrap) and build again." >&2
+    exit 1
+fi
+
+echo "[archive-check $label] $copy_index is the archive at $source_index"

changelog

@@ -1,3 +1,24 @@
+turnkey-nodebb-19.0 (2) turnkey; urgency=low
+
+  * The build proves that the project archive it installs from is the live
+    one. bin/keel-archive-check compares the package index copied into the
+    build tree with the index of the archive on the build host and stops the
+    build when they differ: in bootstrap/post, where the copy is made, and
+    again on the tree that is about to be configured, because fab stamps the
+    bootstrap and a rebuild otherwise reuses the copy an earlier build left.
+
+  * conf.d/zz-project-packages replaces the assertion on a literal version,
+    which passed on a stale index and let a rebuild ship the previous
+    morning's inithooks and confconsole. For each of inithooks, confconsole
+    and keel it checks that the archive offers exactly one version, that
+    apt's candidate is that version, that the version is served by the
+    project archive and not by an upstream source, and that the installed
+    package is that candidate and configured. It then removes the build time
+    package source, which conf.d/main used to do. A rebuild made after a
+    publication now either carries the newly published versions or fails.
+
+ -- Keel Linux maintainers <admin@keellinux.org>  Sun, 27 Sep 2026 00:45:00 +0000
+
 turnkey-nodebb-19.0 (1) turnkey; urgency=low
 
   * Initial release of the NodeBB appliance for Keel, compatible with

conf.d/main

@@ -100,16 +100,8 @@ apt-get install -y --only-upgrade \
     -o Dpkg::Options::=--force-confdef \
     -o Dpkg::Options::=--force-confold \
     inithooks confconsole
-[[ $(dpkg-query -W -f '${Version}\n' inithooks confconsole | grep -c '+keel1$') -eq 2 ]]
-
-# a half configured package keeps its new version, so check the state too
-[[ $(dpkg-query -W -f '${Status}\n' inithooks confconsole keel \
-    | grep -c '^install ok installed$') -eq 3 ]]
 dpkg-query -W -f '${Package} ${Version} ${Status}\n' inithooks confconsole keel
 
-# the build time package source is not for appliances: remove it, keep the
-# documented, disabled entry for the future signed repository (overlay)
-rm -f /etc/apt/sources.list.d/keel-staging.list
-rm -rf /srv/keel-apt
-rm -rf /var/lib/apt/lists/*
-grep -q '^Enabled: no' /etc/apt/sources.list.d/keel.sources
+# conf.d/zz-project-packages runs after this script: it checks these three
+# against the archive the build read and then removes the build time source.
+# No version is asserted here, because a literal one passes on a stale index.

conf.d/zz-project-packages

@@ -0,0 +1,80 @@
+#!/bin/bash -e
+# The project's own packages (inithooks, confconsole, keel) are installed and
+# upgraded by conf.d/main from the build host's staging archive, copied into
+# the build tree. This runs last: it proves that what is installed is what
+# that archive offers, and then takes the archive away, because no appliance
+# ships the build time package source.
+#
+# Nothing here names a version. A literal passes whenever the build happens to
+# read a stale index, which is how the forum layer of 2026-09-26 shipped the
+# previous morning's inithooks and confconsole with a green build. For each
+# package:
+#
+#   1. the archive offers exactly one version of it
+#   2. apt's candidate is that version, so the project archive won over the
+#      upstream one
+#   3. that version is served by the project archive and not by an upstream
+#      source, so the installed package is a project build
+#   4. the installed version is that candidate, and dpkg has it configured
+#
+# So a rebuild made after a publication either carries the newly published
+# versions or fails here.
+#
+# The paths are overridable so tests/project-packages.bats can run this
+# against a scratch tree, the same way the first boot hook is tested.
+
+KEEL_APT_ROOT="${KEEL_APT_ROOT:-/srv/keel-apt}"
+KEEL_APT_REPO="$KEEL_APT_ROOT/repo"
+KEEL_STAGING_LIST="${KEEL_STAGING_LIST:-/etc/apt/sources.list.d/keel-staging.list}"
+KEEL_SOURCES="${KEEL_SOURCES:-/etc/apt/sources.list.d/keel.sources}"
+KEEL_APT_LISTS="${KEEL_APT_LISTS:-/var/lib/apt/lists}"
+KEEL_PROJECT_PACKAGES="${KEEL_PROJECT_PACKAGES:-inithooks confconsole keel}"
+
+fatal() {
+    echo "fatal [zz-project-packages]: $*" >&2
+    exit 1
+}
+
+# the distribution is the one the build wrote into the source list, so this
+# script and the Makefile cannot drift apart
+DIST_OF_LIST='$1 == "deb" { for (i = 2; i <= NF; i++) if ($i ~ /^file:/) { print $(i + 1); exit } }'
+VERSIONS_OF='$1 == "Package:" { name = $2 } $1 == "Version:" && name == want { print $2 }'
+
+dist=$(awk "$DIST_OF_LIST" "$KEEL_STAGING_LIST" 2>/dev/null || true)
+[ -n "$dist" ] || fatal "no file: source in $KEEL_STAGING_LIST: this build had no project archive"
+
+index=$KEEL_APT_REPO/dists/$dist/main/binary-$(dpkg --print-architecture)/Packages
+[ -f "$index" ] || fatal "no package index at $index"
+
+for package in $KEEL_PROJECT_PACKAGES; do
+    mapfile -t offered < <(awk -v want="$package" "$VERSIONS_OF" "$index")
+    [ "${#offered[@]}" -eq 1 ] \
+        || fatal "$package: $index offers ${#offered[@]} versions (${offered[*]:-none}), expected one"
+
+    candidate=$(apt-cache policy "$package" | awk '$1 == "Candidate:" { print $2 }')
+    [ "$candidate" = "${offered[0]}" ] \
+        || fatal "$package: apt would install '$candidate' and the project archive offers '${offered[0]}': the index this build read is not the archive"
+
+    apt-cache madison "$package" \
+        | awk -F' *\\| *' -v version="$candidate" '$2 == version { print $3 }' \
+        | grep -qF "file:$KEEL_APT_REPO" \
+        || fatal "$package $candidate does not come from $KEEL_APT_REPO: it is an upstream build"
+
+    installed=$(dpkg-query -W -f '${Version}' "$package" 2>/dev/null) \
+        || fatal "$package is not installed"
+    [ "$installed" = "$candidate" ] \
+        || fatal "$package $installed is installed and the project archive offers $candidate: this build read a stale index"
+
+    status=$(dpkg-query -W -f '${Status}' "$package")
+    [ "$status" = "install ok installed" ] \
+        || fatal "$package is '$status', not 'install ok installed'"
+
+    echo "$package $installed, the candidate of the project archive"
+done
+
+# the build time package source is not for appliances: remove it, and keep the
+# documented, disabled entry for the future signed repository (overlay)
+rm -f "$KEEL_STAGING_LIST"
+rm -rf "$KEEL_APT_ROOT"
+rm -rf "${KEEL_APT_LISTS:?}"/*
+grep -q '^Enabled: no' "$KEEL_SOURCES"

tests/README.md

@@ -23,6 +23,13 @@ matches the spec.
 - `nodebb.bats`: unit tests of
   `overlay/usr/lib/inithooks/lib/nodebb.sh`, the logic behind the first boot
   hook `40nodebb`.
+- `archive-check.bats`: unit tests of `bin/keel-archive-check`, which the
+  Makefile runs twice per build so that the copy of the project archive inside
+  the build tree is the archive as it is at build time.
+- `project-packages.bats`: unit tests of `conf.d/zz-project-packages`, the
+  last conf script, which checks each project package against that archive.
+  `dpkg`, `dpkg-query` and `apt-cache` are PATH stubs reading fixtures, so no
+  chroot and no apt are needed.
 - `coverage.sh`: runs each bats file under kcov and fails when any measured
   library is below `COVERAGE_THRESHOLD` (default 95).
 - `instance.yaml`: the spec the test container boots from. Not the forum: