The build verifies the staging archive instead of reading it unverified

.github/workflows/tests.yml

@@ -3,6 +3,8 @@
 #              kcov, on hosted runners, threshold committed here
 #   appliance: fetch the published layer, verify it, assemble it, boot it in
 #              LXC and check the forum over IPv6, on the self-hosted runner
+#   package:   a change that the layer ships needs a changelog entry, so it
+#              can reach a machine
 name: tests
 
 on:
@@ -46,3 +48,29 @@ jobs:
       # checks this against the parent the published manifest records.
       parent: nodejs-nginx
       timeout: 60
+  package:
+    # "package / changelog" is a required status on main in this repository's
+    # protection, and until now no job produced it: a check that is required
+    # and never reported leaves every pull request blocked for good. The job
+    # is the one keel-mariadb, keel-postgresql and keel-wordpress carry.
+    #
+    # Only on a pull request. require-changelog compares the two commits of
+    # the pull request, which it reads from
+    # github.event.pull_request.base.sha; outside a pull request that is
+    # empty and the script is handed one argument instead of two, so every
+    # push to main failed the check. A job whose condition is false is
+    # reported as skipped, which satisfies a required status, so protection
+    # keeps working and the check is still a real run on the pull request
+    # itself, which is the only place it can say anything.
+    if: github.event_name == 'pull_request'
+    # This recipe has no debian/changelog: what bt-layer records in the
+    # manifest is the top entry of ./changelog, which is also what
+    # make-release-deb.py turns into the release package. So that is the
+    # file a change which ships has to bump. This job produces the check
+    # "package / changelog".
+    uses: keel-linux/.github/.github/workflows/require-changelog.yml@main
+    with:
+      changelog: changelog
+      # keel/ is the example instance description, which the layer does not
+      # ship; the rest is the reusable workflow's own default.
+      exempt: '^(tests/|docs/|\.github/|keel/|README|COVERAGE\.md|LICENSE|\.gitignore)'

COVERAGE.md

@@ -4,22 +4,22 @@ Standard: decisions 0003 (90 percent per repository, 95 for code the project
 writes) and 0004 (bats plus kcov for shell; a build and a boot on LXC as the
 acceptance test of an appliance recipe, docs/org-plan.md section 1).
 
-## Measured 2026-09-26
+## Measured 2026-09-27
 
 | File | Test | Lines | Note |
 | --- | --- | --- | --- |
 | overlay/usr/lib/inithooks/lib/nodebb.sh | tests/nodebb.bats (25 tests) | 100 percent (43/43) under kcov | every function and every branch |
 | overlay/usr/lib/inithooks/firstboot.d/40nodebb | tests/hook.bats (15 tests) | 96.97 percent (32/33) under kcov | the one uncovered line is inside the dialog loop, which needs a terminal |
 | overlay/etc/nginx/* | tests/nginx.bats (12 tests) | not executable | asserted as content: the geo variable, the map, the listeners, the proxy headers |
 | tests/lib/boot-test-lib.sh | tests/boot-test.bats (41 tests) | 100 percent (137/137) under kcov | the logic of the boot test: argument parsing, address discovery, deadlines, the HTTP and diff verdicts |
-| bin/keel-archive-check | tests/archive-check.bats (8 tests) | 100 percent (26/26) under kcov | the build time check that the archive copy in the build tree is the live archive |
-| conf.d/zz-project-packages | tests/project-packages.bats (13 tests) | 100 percent (29/29) under kcov | the build time check that each project package is the candidate of the archive, and a project build |
+| bin/keel-archive-check | tests/archive-check.bats (25 tests) | 100 percent (52/52) under kcov | the build time check: the archive copy in the build tree is the live archive, the source entry names the keyring through signed-by, nothing says trusted=yes, and the signature on the copied InRelease verifies against the staging key (tracker#7) |
+| conf.d/zz-project-packages | tests/project-packages.bats (14 tests) | 100 percent (31/31) under kcov | the build time check that each project package is the candidate of the archive and a project build, and that the archive copy, its source entry and the staging keyring leave the image |
 | overlay/usr/lib/inithooks/bin/nodebb.py | none | 0 | dialog wrapper, only reached with a terminal attached |
 | conf.d/main | tests/boot-test.sh (build step) | integration only | build time script, 0004 pragmatic limits |
 | tests/boot-test.sh | itself | integration only | the thin main of the acceptance test: keel and LXC as root |
 
-Total over the five measured shell files: 99.63 percent (267/268) before the
-terminal test, 100 percent (268/268) with it.
+Total over the five measured shell files: 99.66 percent (295/296) before the
+terminal test, 100 percent (296/296) with it, over 132 bats tests.
 
 `tests/coverage.sh` runs the whole bats suite under kcov, measures the
 library, the first boot hook, the boot test's own library and the two build

Makefile

@@ -10,24 +10,54 @@ include $(FAB_PATH)/common/mk/turnkey.mk
 
 # The project's own packages (inithooks, confconsole, keel) come from the
 # build host's APT repository during the build only. The repository is copied
-# into the bootstrap and listed as a [trusted=yes] file source, because the
-# staging distribution is unsigned; conf.d/zz-project-packages checks what was
-# installed against that copy, removes both from the image and leaves the
-# future apt.keellinux.org entry in place, disabled.
+# into the bootstrap and the build verifies it there, the way an appliance
+# verifies the release archive (tracker#7): the public half of the staging key
+# is installed as a keyring, the source entry names it through signed-by,
+# nothing in the tree says trusted=yes, and apt runs with --error-on=any, so a
+# signature that cannot be checked fails the build instead of warning about it
+# and carrying on. conf.d/zz-project-packages checks what was installed
+# against that copy, then removes the copy, the source entry and the keyring
+# from the image and leaves the future apt.keellinux.org entry in place,
+# disabled.
+#
+# None of the three build time files is for an installed appliance, because
+# the staging key signs whatever the build host produced. The removelist at
+# common/removelists-final/turnkey takes all three out of the image as well,
+# whatever a recipe does. keel-mariadb, keel-postgresql and keel-wordpress carry this block too,
+# and tracker#7 is the second defect that had to be fixed in all four: it
+# belongs in the shared tree, next to the removelist that undoes it.
 KEEL_APT_REPO ?= /srv/keel-apt/repo
 KEEL_APT_DIST ?= trixie-staging
-KEEL_ARCHIVE_CHECK = $(CURDIR)/bin/keel-archive-check $(KEEL_APT_REPO)
+# Beside the repository rather than inside it: bin/publish of keel-linux/apt
+# installs the public half of whichever key it signed a distribution with
+# here, so the key a build verifies with cannot drift from the key the archive
+# was signed with.
+KEEL_APT_KEYRING ?= /srv/keel-apt/keys/keel-staging-keyring.asc
+# Where that key goes in the build tree, and which key has to be in it: the
+# staging signing subkey (handbook decision 0011). A keyring is only a promise
+# until the key inside it is named, so bin/keel-archive-check fails the build
+# when the keyring it finds holds some other key.
+KEEL_APT_KEYRING_PATH ?= /etc/apt/keyrings/keel-staging-keyring.asc
+KEEL_APT_KEY ?= 8CFD1A4841448B2227341CEB202CACBD0E97090A
+KEEL_STAGING_LIST ?= /etc/apt/sources.list.d/keel-staging.list
+KEEL_ARCHIVE_CHECK = KEEL_ARCHIVE_KEY=$(KEEL_APT_KEY) \
+	KEEL_ARCHIVE_KEYRING=$(KEEL_APT_KEYRING_PATH) \
+	KEEL_ARCHIVE_LIST=$(KEEL_STAGING_LIST) \
+	$(CURDIR)/bin/keel-archive-check $(KEEL_APT_REPO)
 
 # The copy is made fresh and then proved: bin/keel-archive-check compares the
-# copied package index with the live one and stops the build when they differ.
+# copied package index with the live one, verifies the signature on the copied
+# InRelease against the keyring, refuses any trusted=yes, and stops the build
+# when one of them is wrong.
 define _keel_bootstrap/post
 
-	mkdir -p $O/bootstrap/srv/keel-apt/repo;
+	mkdir -p $O/bootstrap/srv/keel-apt/repo $O/bootstrap$(dir $(KEEL_APT_KEYRING_PATH));
 	rm -rf $O/bootstrap/srv/keel-apt/repo/dists $O/bootstrap/srv/keel-apt/repo/pool;
 	cp -a $(KEEL_APT_REPO)/dists $(KEEL_APT_REPO)/pool $O/bootstrap/srv/keel-apt/repo/;
+	install -m 644 $(KEEL_APT_KEYRING) $O/bootstrap$(KEEL_APT_KEYRING_PATH);
+	echo "deb [signed-by=$(KEEL_APT_KEYRING_PATH)] file:///srv/keel-apt/repo $(KEEL_APT_DIST) main" > $O/bootstrap$(KEEL_STAGING_LIST);
 	$(KEEL_ARCHIVE_CHECK) $O/bootstrap $(KEEL_APT_DIST) $(FAB_ARCH) bootstrap;
-	echo "deb [trusted=yes] file:///srv/keel-apt/repo $(KEEL_APT_DIST) main" > $O/bootstrap/etc/apt/sources.list.d/keel-staging.list;
-	fab-chroot $O/bootstrap "apt-get update";
+	fab-chroot $O/bootstrap "apt-get update --error-on=any";
 endef
 bootstrap/post += $(_keel_bootstrap/post)
 
@@ -36,7 +66,9 @@ bootstrap/post += $(_keel_bootstrap/post)
 # all. On 2026-09-26 "make clean" failed on a busy deck, the stamps survived,
 # and the rebuild installed the packages the archive had held that morning
 # without a word. So the tree that is about to be configured is checked on
-# every build, whether or not this build made the bootstrap.
+# every build, whether or not this build made the bootstrap. That check
+# verifies the signature with gpgv too, which is what proves this tree at a
+# step where no apt-get update runs.
 define _keel_root.patched/pre
 
 	$(KEEL_ARCHIVE_CHECK) $O/root.patched $(KEEL_APT_DIST) $(FAB_ARCH) root.patched;

README.rst

@@ -57,15 +57,25 @@ appliance.
 
 The project's own packages (inithooks, confconsole, keel) are listed in the
 plan and resolved, during the build only, from the build host's repository
-copied into the bootstrap as a ``[trusted=yes] file:///srv/keel-apt/repo``
-source (``Makefile``, ``bootstrap/post``).
-
-That copy has to be the archive as it is at build time: fab stamps the
+copied into the bootstrap as a ``file:///srv/keel-apt/repo`` source
+(``Makefile``, ``bootstrap/post``).
+
+The build verifies that archive the way an appliance verifies the release one.
+The public half of the staging key is installed into the build tree as
+``/etc/apt/keyrings/keel-staging-keyring.asc``, the source entry names it
+through ``signed-by``, and ``apt-get update`` runs with ``--error-on=any``, so
+a signature that cannot be checked fails the build. It used to say
+``[trusted=yes]``, which switches verification off: apt then printed
+``W: OpenPGP signature verification failed ... Missing key`` and installed the
+packages anyway (tracker#7).
+
+That copy also has to be the archive as it is at build time: fab stamps the
 bootstrap target, so a rebuild would otherwise reuse the copy an earlier build
 made and install packages the archive no longer offers.
-``bin/keel-archive-check`` compares the copied package index with the live one
-where the copy is made, and again on the tree that is about to be configured,
-and stops the build when they differ.
+``bin/keel-archive-check`` compares the copied package index with the live one,
+verifies the signature on the copied ``InRelease`` against that keyring,
+refuses any ``trusted=yes`` anywhere in the tree, and does all of it twice:
+where the copy is made, and again on the tree that is about to be configured.
 
 ``conf.d/zz-project-packages`` runs last. For each project package it checks
 that the archive offers exactly one version, that apt's candidate is that

bin/keel-archive-check

@@ -1,13 +1,30 @@
 #!/bin/bash
-# The build reads the project's APT archive from a copy inside the build tree.
-# This checks that the copy is the archive as it is right now, byte for byte,
-# and stops the build when it is not.
+# The build reads the project's own packages from a copy of the project's APT
+# archive inside the build tree. This checks two things about that copy and
+# stops the build when either one is wrong:
 #
-# Why it exists: fab stamps the bootstrap target, so a second build of the same
-# product reuses the bootstrap the first one made, copy of the archive and all.
-# On 2026-09-26 "make clean" failed on a busy deck, the stamps survived, and
-# the rebuild installed the packages the archive had held that morning. The
-# build said nothing, because every step of it succeeded.
+#   1. the copy is the archive as it is right now, byte for byte;
+#   2. the build's apt can verify the copy, and is not being told not to.
+#
+# Why (1) is checked: fab stamps the bootstrap target, so a second build of
+# the same product reuses the bootstrap the first one made, copy of the
+# archive and all. On 2026-09-26 "make clean" failed on a busy deck, the
+# stamps survived, and the rebuild installed the packages the archive had held
+# that morning. The build said nothing, because every step of it succeeded.
+#
+# Why (2) is checked (tracker#7): the staging distribution was given its own
+# signing key and the build was never given the public half, while the source
+# entry said [trusted=yes]. So apt printed
+#
+#   W: OpenPGP signature verification failed: file:/srv/keel-apt/repo
+#      trixie-staging InRelease: Missing key 8CFD...090A
+#
+# and installed the project's packages unverified, which is a warning nobody
+# fails on. Now the keyring, the key in it, the signed-by option and the
+# absence of any trusted=yes are all checked here, and the copied InRelease is
+# verified with gpgv against that keyring before a package is installed from
+# it. gpgv rather than apt's word for it, so the tree that is about to be
+# configured is proved even at a step where no apt-get update runs.
 #
 #   keel-archive-check SOURCE_REPO TREE DIST ARCH [LABEL]
 #
@@ -17,11 +34,24 @@
 # ARCH         the Debian architecture, e.g. amd64
 # LABEL        name of the build step, for the messages (default: TREE)
 #
-# Exit 0 when the copy matches, 1 otherwise.
+# Read from the environment, so the Makefile block stays one line per step:
+#
+# KEEL_ARCHIVE_KEY      fingerprint of the key that must sign the archive,
+#                       with no spaces. Required: a keyring is only a
+#                       promise until the key inside it is named.
+# KEEL_ARCHIVE_KEYRING  where in TREE the keyring is
+# KEEL_ARCHIVE_LIST     where in TREE the source entry is
+#
+# Exit 0 when everything holds, 1 otherwise.
 set -eu
 
+KEEL_ARCHIVE_KEY="${KEEL_ARCHIVE_KEY:-}"
+KEEL_ARCHIVE_KEYRING="${KEEL_ARCHIVE_KEYRING:-/etc/apt/keyrings/keel-staging-keyring.asc}"
+KEEL_ARCHIVE_LIST="${KEEL_ARCHIVE_LIST:-/etc/apt/sources.list.d/keel-staging.list}"
+
 usage() {
     echo "usage: $(basename "$0") SOURCE_REPO TREE DIST ARCH [LABEL]" >&2
+    echo "environment: KEEL_ARCHIVE_KEY (required), KEEL_ARCHIVE_KEYRING, KEEL_ARCHIVE_LIST" >&2
     exit 1
 }
 
@@ -36,15 +66,23 @@ for value in "$source_repo" "$tree" "$dist" "$arch"; do
     [ -n "$value" ] || usage
 done
 
-index=dists/$dist/main/binary-$arch/Packages
-source_index=$source_repo/$index
-copy_index=$tree/srv/keel-apt/repo/$index
-
 fatal() {
     echo "FATAL [archive-check $label]: $*" >&2
     exit 1
 }
 
+[ -n "$KEEL_ARCHIVE_KEY" ] \
+    || fatal "KEEL_ARCHIVE_KEY names no key: the archive would be verified against whatever the keyring happens to hold"
+
+index=dists/$dist/main/binary-$arch/Packages
+source_index=$source_repo/$index
+copy=$tree/srv/keel-apt/repo
+copy_index=$copy/$index
+keyring=$tree/$KEEL_ARCHIVE_KEYRING
+list=$tree/$KEEL_ARCHIVE_LIST
+inrelease=$copy/dists/$dist/InRelease
+
+# 1. The copy is the archive as it is right now.
 [ -f "$source_index" ] || fatal "the archive has no index at $source_index"
 [ -f "$copy_index" ] || fatal "the build tree has no archive index at $copy_index"
 
@@ -57,4 +95,48 @@ if ! cmp -s "$source_index" "$copy_index"; then
     exit 1
 fi
 
+# 2. Nothing in the tree switches verification off. A single trusted=yes
+# anywhere turns every failure below into a warning, which is the defect this
+# check exists for, so it is refused wherever it is written, in either of the
+# two formats apt reads a source in.
+verification_off='trusted *= *yes|^ *Trusted: *yes'
+apt_sources="$tree/etc/apt/sources.list $tree/etc/apt/sources.list.d"
+# shellcheck disable=SC2086 # the two paths are one word each, on purpose
+untrusted=$(grep -rlE "$verification_off" $apt_sources 2>/dev/null || true)
+[ -z "$untrusted" ] \
+    || fatal "verification is switched off in: $(echo "$untrusted" | tr '\n' ' ')"
+
+# 3. The source entry names this distribution and the keyring it is verified
+# with. apt takes signed-by from the entry, so the entry is what decides
+# whether anything is verified at all.
+[ -f "$list" ] || fatal "the build tree has no source entry at $list"
+entry=$(grep -E "^[[:space:]]*deb[[:space:]]" "$list" | head -1)
+[ -n "$entry" ] || fatal "$list has no deb line"
+[[ "$entry" == *"signed-by=$KEEL_ARCHIVE_KEYRING"* ]] \
+    || fatal "$list does not name signed-by=$KEEL_ARCHIVE_KEYRING: $entry"
+# Padded with spaces on both sides so that trixie does not pass for the entry
+# of trixie-staging.
+[[ " $entry " == *" $dist "* ]] \
+    || fatal "$list does not name the distribution $dist: $entry"
+
+# 4. The keyring is there and holds the key that must have signed the archive.
+# A keyring that is present but holds the wrong key reads as a configured
+# build and fails only at apt, in a line that scrolls past.
+[ -s "$keyring" ] || fatal "$keyring is missing or empty: the build cannot verify the archive"
+gpg --batch --show-keys --with-colons "$keyring" 2>/dev/null \
+    | awk -F: -v want="$KEEL_ARCHIVE_KEY" '$1 == "fpr" && $10 == want { found = 1 } END { exit !found }' \
+    || fatal "$keyring does not hold key $KEEL_ARCHIVE_KEY"
+
+# 5. The signature on the copied index is good under that keyring. gpgv wants
+# a binary keyring, and dearmouring a public key needs no agent and no
+# network.
+[ -s "$inrelease" ] || fatal "$inrelease is missing or empty: this copy of the archive is not signed"
+binary_keyring=$(mktemp)
+trap 'rm -f "$binary_keyring"' EXIT
+if ! gpg --batch --dearmor < "$keyring" > "$binary_keyring" 2>/dev/null \
+    || ! gpgv --keyring "$binary_keyring" "$inrelease" >/dev/null 2>&1; then
+    fatal "the signature on $inrelease does not verify against $keyring"
+fi
+
 echo "[archive-check $label] $copy_index is the archive at $source_index"
+echo "[archive-check $label] $inrelease verifies against $KEEL_ARCHIVE_KEY, nothing is trusted unverified"