The build verifies the staging archive instead of reading it unverified

COVERAGE.md

@@ -10,26 +10,28 @@ acceptance test of a recipe, docs/org-plan.md section 1).
 | --- | --- | --- | --- |
 | `overlay/usr/lib/inithooks/lib/wordpress.sh` | `tests/wordpress.bats` (40 tests) | 99.00 percent (99/100) under kcov | every function and every branch |
 | `overlay/usr/lib/inithooks/firstboot.d/40wordpress` | `tests/hook.bats` (29 tests) | 97.73 percent (43/44) under kcov | the hook itself, run for real |
-| `tests/lib/boot-test-lib.sh` | `tests/boot-test.bats` (66 tests) | 98.88 percent (264/267) under kcov | parsing, addresses, deadlines, the container marks, every verdict |
-| `conf.d/zzz-keel-archive` | `tests/keel-archive.bats` (12 tests) | 100 percent (24/24) under kcov | every way it enables and every way it refuses |
-| `conf.d/zz-project-packages` | `tests/project-packages.bats` (13 tests) | 100 percent (29/29) under kcov | shared with keel-nodebb, where the pattern is maintained |
-| `bin/keel-archive-check` | `tests/archive-check.bats` (8 tests) | 100 percent (26/26) under kcov | same |
+| `tests/lib/boot-test-lib.sh` | `tests/boot-test.bats` (73 tests) | 98.95 percent (282/285) under kcov | parsing, addresses, deadlines, the container marks, every verdict, and the image carrying none of the build time archive files |
+| `conf.d/zzz-keel-archive` | `tests/keel-archive.bats` (13 tests) | 100 percent (26/26) under kcov | every way it enables and every way it refuses, including a staging keyring left in the image |
+| `conf.d/zz-project-packages` | `tests/project-packages.bats` (14 tests) | 100 percent (31/31) under kcov | shared with keel-nodebb, where the pattern is maintained |
+| `bin/keel-archive-check` | `tests/archive-check.bats` (27 tests) | 100 percent (54/54) under kcov | the build time check of tracker#7: the copy is the live archive, the entry names the keyring through signed-by, nothing says trusted=yes, and the copied InRelease verifies against the staging key |
 | `overlay/usr/lib/inithooks/bin/wordpress.py` | none | 0 | dialog wrapper, only reached with a terminal attached |
 | `overlay/usr/lib/inithooks/lib/*.php` | the boot test | integration only | two PHP files `wp eval-file` runs; `conf.d/main` has PHP lint them |
 | `conf.d/main` | the build | integration only | build time script, 0004 pragmatic limits |
 | `tests/boot-test.sh` | itself | integration only | the thin main of the acceptance test: keel and LXC as root |
 
-Total over the six measured shell files: **98.98 percent (485/490)**, 168 bats
+Total over the six measured shell files: **99.07 percent (535/540)**, 196 bats
 tests, none failing. `tests/coverage.sh` fails below `COVERAGE_THRESHOLD`, which the workflow
 sets to **97**, the lowest measured file. It is only ever raised (decision
 0006).
 
     $ COVERAGE_THRESHOLD=97 tests/coverage.sh
     kcov line coverage (threshold 97 percent):
-      99.00  99/100  wordpress.sh
-     100.00  24/24  zzz-keel-archive
+     100.00  31/31  zz-project-packages
+     100.00  26/26  zzz-keel-archive
       97.73  43/44  40wordpress
-      98.88  264/267  boot-test-lib.sh
+      98.95  282/285  boot-test-lib.sh
+      99.00  99/100  wordpress.sh
+     100.00  54/54  keel-archive-check
      100.00  29/29  zz-project-packages
      100.00  26/26  keel-archive-check
 

Makefile

@@ -37,28 +37,54 @@ COMMON_OVERLAYS += $(CURDIR)/overlay
 
 include $(FAB_PATH)/common/mk/turnkey.mk
 
-# The project's own packages (inithooks, confconsole, keel and, new here,
-# keel-archive-keyring) come from the build host's APT repository during the
-# build only. The repository is copied into the bootstrap and listed as a
-# [trusted=yes] file source, because the staging distribution is unsigned;
-# conf.d/zz-project-packages checks what was installed against that copy and
-# removes both from the image, and conf.d/zzz-keel-archive then enables the
-# signed repository the appliance uses at run time. Same block as
-# keel-nodebb, which is where the pattern is maintained.
+# The project's own packages (inithooks, confconsole, keel and keel-archive-
+# keyring) come from the build host's APT repository during the build only.
+# The repository is copied into the bootstrap and the build verifies it there,
+# the way an appliance verifies the release archive (tracker#7): the public
+# half of the staging key is installed as a keyring, the source entry names it
+# through signed-by, nothing in the tree says trusted=yes, and apt runs with
+# --error-on=any, so a signature that cannot be checked fails the build
+# instead of warning about it and carrying on. conf.d/zz-project-packages
+# checks what was installed against that copy and removes the copy, the source
+# entry and the keyring from the image, and conf.d/zzz-keel-archive then
+# enables the signed repository the appliance uses at run time.
+#
+# None of the three build time files is for an installed appliance, because
+# the staging key signs whatever the build host produced. The removelist at
+# common/removelists-final/turnkey takes all three out of the image as well,
+# whatever a recipe does. Same block as keel-nodebb, which is where the pattern is maintained.
 KEEL_APT_REPO ?= /srv/keel-apt/repo
 KEEL_APT_DIST ?= trixie-staging
-KEEL_ARCHIVE_CHECK = $(CURDIR)/bin/keel-archive-check $(KEEL_APT_REPO)
+# Beside the repository rather than inside it: bin/publish of keel-linux/apt
+# installs the public half of whichever key it signed a distribution with
+# here, so the key a build verifies with cannot drift from the key the archive
+# was signed with.
+KEEL_APT_KEYRING ?= /srv/keel-apt/keys/keel-staging-keyring.asc
+# Where that key goes in the build tree, and which key has to be in it: the
+# staging signing subkey (handbook decision 0011). A keyring is only a promise
+# until the key inside it is named, so bin/keel-archive-check fails the build
+# when the keyring it finds holds some other key.
+KEEL_APT_KEYRING_PATH ?= /etc/apt/keyrings/keel-staging-keyring.asc
+KEEL_APT_KEY ?= 8CFD1A4841448B2227341CEB202CACBD0E97090A
+KEEL_STAGING_LIST ?= /etc/apt/sources.list.d/keel-staging.list
+KEEL_ARCHIVE_CHECK = KEEL_ARCHIVE_KEY=$(KEEL_APT_KEY) \
+	KEEL_ARCHIVE_KEYRING=$(KEEL_APT_KEYRING_PATH) \
+	KEEL_ARCHIVE_LIST=$(KEEL_STAGING_LIST) \
+	$(CURDIR)/bin/keel-archive-check $(KEEL_APT_REPO)
 
 # The copy is made fresh and then proved: bin/keel-archive-check compares the
-# copied package index with the live one and stops the build when they differ.
+# copied package index with the live one, verifies the signature on the copied
+# InRelease against the keyring, refuses any trusted=yes, and stops the build
+# when one of them is wrong.
 define _keel_bootstrap/post
 
-	mkdir -p $O/bootstrap/srv/keel-apt/repo;
+	mkdir -p $O/bootstrap/srv/keel-apt/repo $O/bootstrap$(dir $(KEEL_APT_KEYRING_PATH));
 	rm -rf $O/bootstrap/srv/keel-apt/repo/dists $O/bootstrap/srv/keel-apt/repo/pool;
 	cp -a $(KEEL_APT_REPO)/dists $(KEEL_APT_REPO)/pool $O/bootstrap/srv/keel-apt/repo/;
+	install -m 644 $(KEEL_APT_KEYRING) $O/bootstrap$(KEEL_APT_KEYRING_PATH);
+	echo "deb [signed-by=$(KEEL_APT_KEYRING_PATH)] file:///srv/keel-apt/repo $(KEEL_APT_DIST) main" > $O/bootstrap$(KEEL_STAGING_LIST);
 	$(KEEL_ARCHIVE_CHECK) $O/bootstrap $(KEEL_APT_DIST) $(FAB_ARCH) bootstrap;
-	echo "deb [trusted=yes] file:///srv/keel-apt/repo $(KEEL_APT_DIST) main" > $O/bootstrap/etc/apt/sources.list.d/keel-staging.list;
-	fab-chroot $O/bootstrap "apt-get update";
+	fab-chroot $O/bootstrap "apt-get update --error-on=any";
 endef
 bootstrap/post += $(_keel_bootstrap/post)
 
@@ -67,7 +93,9 @@ bootstrap/post += $(_keel_bootstrap/post)
 # all. On 2026-09-26 "make clean" failed on a busy deck, the stamps survived,
 # and the rebuild installed the packages the archive had held that morning
 # without a word. So the tree that is about to be configured is checked on
-# every build, whether or not this build made the bootstrap.
+# every build, whether or not this build made the bootstrap. That check
+# verifies the signature with gpgv too, which is what proves this tree at a
+# step where no apt-get update runs.
 define _keel_root.patched/pre
 
 	$(KEEL_ARCHIVE_CHECK) $O/root.patched $(KEEL_APT_DIST) $(FAB_ARCH) root.patched;

bin/keel-archive-check

@@ -1,13 +1,30 @@
 #!/bin/bash
-# The build reads the project's APT archive from a copy inside the build tree.
-# This checks that the copy is the archive as it is right now, byte for byte,
-# and stops the build when it is not.
+# The build reads the project's own packages from a copy of the project's APT
+# archive inside the build tree. This checks two things about that copy and
+# stops the build when either one is wrong:
 #
-# Why it exists: fab stamps the bootstrap target, so a second build of the same
-# product reuses the bootstrap the first one made, copy of the archive and all.
-# On 2026-09-26 "make clean" failed on a busy deck, the stamps survived, and
-# the rebuild installed the packages the archive had held that morning. The
-# build said nothing, because every step of it succeeded.
+#   1. the copy is the archive as it is right now, byte for byte;
+#   2. the build's apt can verify the copy, and is not being told not to.
+#
+# Why (1) is checked: fab stamps the bootstrap target, so a second build of
+# the same product reuses the bootstrap the first one made, copy of the
+# archive and all. On 2026-09-26 "make clean" failed on a busy deck, the
+# stamps survived, and the rebuild installed the packages the archive had held
+# that morning. The build said nothing, because every step of it succeeded.
+#
+# Why (2) is checked (tracker#7): the staging distribution was given its own
+# signing key and the build was never given the public half, while the source
+# entry said [trusted=yes]. So apt printed
+#
+#   W: OpenPGP signature verification failed: file:/srv/keel-apt/repo
+#      trixie-staging InRelease: Missing key 8CFD...090A
+#
+# and installed the project's packages unverified, which is a warning nobody
+# fails on. Now the keyring, the key in it, the signed-by option and the
+# absence of any trusted=yes are all checked here, and the copied InRelease is
+# verified with gpgv against that keyring before a package is installed from
+# it. gpgv rather than apt's word for it, so the tree that is about to be
+# configured is proved even at a step where no apt-get update runs.
 #
 #   keel-archive-check SOURCE_REPO TREE DIST ARCH [LABEL]
 #
@@ -17,11 +34,26 @@
 # ARCH         the Debian architecture, e.g. amd64
 # LABEL        name of the build step, for the messages (default: TREE)
 #
-# Exit 0 when the copy matches, 1 otherwise.
+# Read from the environment, so the Makefile block stays one line per step:
+#
+# KEEL_ARCHIVE_KEY      fingerprint of the key that must sign the archive,
+#                       with no spaces. Required: a keyring is only a
+#                       promise until the key inside it is named.
+# KEEL_ARCHIVE_KEYRING  where in TREE the keyring is
+# KEEL_ARCHIVE_LIST     where in TREE the source entry is
+# KEEL_ARCHIVE_PATH     the path an apt source names this archive by, which is
+#                       the archive a trusted=yes is refused for
+#
+# Exit 0 when everything holds, 1 otherwise.
 set -eu
 
+KEEL_ARCHIVE_KEY="${KEEL_ARCHIVE_KEY:-}"
+KEEL_ARCHIVE_KEYRING="${KEEL_ARCHIVE_KEYRING:-/etc/apt/keyrings/keel-staging-keyring.asc}"
+KEEL_ARCHIVE_LIST="${KEEL_ARCHIVE_LIST:-/etc/apt/sources.list.d/keel-staging.list}"
+
 usage() {
     echo "usage: $(basename "$0") SOURCE_REPO TREE DIST ARCH [LABEL]" >&2
+    echo "environment: KEEL_ARCHIVE_KEY (required), KEEL_ARCHIVE_KEYRING, KEEL_ARCHIVE_LIST" >&2
     exit 1
 }
 
@@ -36,15 +68,26 @@ for value in "$source_repo" "$tree" "$dist" "$arch"; do
     [ -n "$value" ] || usage
 done
 
-index=dists/$dist/main/binary-$arch/Packages
-source_index=$source_repo/$index
-copy_index=$tree/srv/keel-apt/repo/$index
-
 fatal() {
     echo "FATAL [archive-check $label]: $*" >&2
     exit 1
 }
 
+[ -n "$KEEL_ARCHIVE_KEY" ] \
+    || fatal "KEEL_ARCHIVE_KEY names no key: the archive would be verified against whatever the keyring happens to hold"
+
+# The path an apt source names this archive by, which is what makes a
+# trusted=yes elsewhere in the tree somebody else's business.
+ARCHIVE_PATH="${KEEL_ARCHIVE_PATH:-/srv/keel-apt/repo}"
+index=dists/$dist/main/binary-$arch/Packages
+source_index=$source_repo/$index
+copy=$tree/srv/keel-apt/repo
+copy_index=$copy/$index
+keyring=$tree/$KEEL_ARCHIVE_KEYRING
+list=$tree/$KEEL_ARCHIVE_LIST
+inrelease=$copy/dists/$dist/InRelease
+
+# 1. The copy is the archive as it is right now.
 [ -f "$source_index" ] || fatal "the archive has no index at $source_index"
 [ -f "$copy_index" ] || fatal "the build tree has no archive index at $copy_index"
 
@@ -57,4 +100,55 @@ if ! cmp -s "$source_index" "$copy_index"; then
     exit 1
 fi
 
+# 2. Nothing in the tree switches verification off for this archive. A single
+# trusted=yes on it turns every failure below into a warning, which is the
+# defect this check exists for, so it is refused in either of the two formats
+# apt reads a source in and in whichever file it is written.
+#
+# Scoped to the sources that name this archive, not to every source in the
+# tree: the captured pool of decision 0012 sets Trusted: yes on purpose, for a
+# file: index generated on this machine from files keel-pool verify checks
+# against the same digests apt does. Refusing that would fail every pinned
+# build. What the pool does is the pool's business; this archive is verified.
+verification_off='trusted *= *yes|^ *Trusted: *yes'
+apt_sources="$tree/etc/apt/sources.list $tree/etc/apt/sources.list.d"
+# shellcheck disable=SC2086 # the two paths are one word each, on purpose
+distrusting=$(grep -rlE "$verification_off" $apt_sources 2>/dev/null || true)
+untrusted=$(echo "$distrusting" | xargs -r grep -lF "$ARCHIVE_PATH" || true)
+[ -z "$untrusted" ] \
+    || fatal "verification is switched off for $ARCHIVE_PATH in: $(echo "$untrusted" | tr '\n' ' ')"
+
+# 3. The source entry names this distribution and the keyring it is verified
+# with. apt takes signed-by from the entry, so the entry is what decides
+# whether anything is verified at all.
+[ -f "$list" ] || fatal "the build tree has no source entry at $list"
+entry=$(grep -E "^[[:space:]]*deb[[:space:]]" "$list" | head -1)
+[ -n "$entry" ] || fatal "$list has no deb line"
+[[ "$entry" == *"signed-by=$KEEL_ARCHIVE_KEYRING"* ]] \
+    || fatal "$list does not name signed-by=$KEEL_ARCHIVE_KEYRING: $entry"
+# Padded with spaces on both sides so that trixie does not pass for the entry
+# of trixie-staging.
+[[ " $entry " == *" $dist "* ]] \
+    || fatal "$list does not name the distribution $dist: $entry"
+
+# 4. The keyring is there and holds the key that must have signed the archive.
+# A keyring that is present but holds the wrong key reads as a configured
+# build and fails only at apt, in a line that scrolls past.
+[ -s "$keyring" ] || fatal "$keyring is missing or empty: the build cannot verify the archive"
+gpg --batch --show-keys --with-colons "$keyring" 2>/dev/null \
+    | awk -F: -v want="$KEEL_ARCHIVE_KEY" '$1 == "fpr" && $10 == want { found = 1 } END { exit !found }' \
+    || fatal "$keyring does not hold key $KEEL_ARCHIVE_KEY"
+
+# 5. The signature on the copied index is good under that keyring. gpgv wants
+# a binary keyring, and dearmouring a public key needs no agent and no
+# network.
+[ -s "$inrelease" ] || fatal "$inrelease is missing or empty: this copy of the archive is not signed"
+binary_keyring=$(mktemp)
+trap 'rm -f "$binary_keyring"' EXIT
+if ! gpg --batch --dearmor < "$keyring" > "$binary_keyring" 2>/dev/null \
+    || ! gpgv --keyring "$binary_keyring" "$inrelease" >/dev/null 2>&1; then
+    fatal "the signature on $inrelease does not verify against $keyring"
+fi
+
 echo "[archive-check $label] $copy_index is the archive at $source_index"
+echo "[archive-check $label] $inrelease verifies against $KEEL_ARCHIVE_KEY, nothing is trusted unverified"

changelog

@@ -1,3 +1,42 @@
+turnkey-wordpress-19.0 (3) turnkey; urgency=low
+
+  * The build verifies the project's own APT archive instead of reading it
+    unverified. The staging distribution has been signed since 2026-09-27, but
+    the build environment had no copy of the public key and the source entry
+    said [trusted=yes], which switches verification off: apt printed
+    "W: OpenPGP signature verification failed ... Missing key
+    8CFD1A4841448B2227341CEB202CACBD0E97090A" and installed inithooks,
+    confconsole and keel anyway. The public half of the staging key is now
+    installed into the build tree as
+    /etc/apt/keyrings/keel-staging-keyring.asc, the source entry names it
+    through signed-by, and apt-get update runs with --error-on=any, so a
+    signature that cannot be checked fails the build instead of warning
+    (tracker#7).
+
+  * bin/keel-archive-check does the same checks itself rather than trusting
+    apt to have complained: the copied package index is the live one, the
+    source entry names the keyring and this distribution, no apt source in
+    the tree says trusted=yes, and the signature on the copied InRelease
+    verifies against the named key with gpgv. It runs where the copy is made
+    and again on the tree that is about to be configured, which is a step
+    where no apt-get update runs at all.
+
+  * The trusted=yes that bin/keel-archive-check refuses is one on the project
+    archive, not one on every apt source in the build tree. The captured pool
+    of decision 0012 sets Trusted: yes on purpose, for a file: index generated
+    on this machine from files keel-pool verify checks against the same digests
+    apt does, so the wider rule would have failed every pinned build.
+
+  * conf.d/zz-project-packages removes the keyring with the source entry and
+    the copy of the archive, conf.d/zzz-keel-archive refuses to enable the
+    appliance's own source while either is still in the image, and the boot
+    test reads the finished rootfs and fails if any of the three is there. The
+    staging key signs whatever the build host produced, so an image that kept
+    it would carry trust in a nightly. The removelist
+    common/removelists-final/turnkey takes all three out as well.
+
+ -- Keel Linux maintainers <admin@keellinux.org>  Sun, 27 Sep 2026 19:30:00 +0000
+
 turnkey-wordpress-19.0 (2) turnkey; urgency=low
 
   * The appliance is a Keel layer on the published mariadb layer: Apache, PHP

conf.d/zz-project-packages

@@ -26,6 +26,7 @@
 KEEL_APT_ROOT="${KEEL_APT_ROOT:-/srv/keel-apt}"
 KEEL_APT_REPO="$KEEL_APT_ROOT/repo"
 KEEL_STAGING_LIST="${KEEL_STAGING_LIST:-/etc/apt/sources.list.d/keel-staging.list}"
+KEEL_STAGING_KEYRING="${KEEL_STAGING_KEYRING:-/etc/apt/keyrings/keel-staging-keyring.asc}"
 KEEL_SOURCES="${KEEL_SOURCES:-/etc/apt/sources.list.d/keel.sources}"
 KEEL_APT_LISTS="${KEEL_APT_LISTS:-/var/lib/apt/lists}"
 KEEL_PROJECT_PACKAGES="${KEEL_PROJECT_PACKAGES:-inithooks confconsole keel}"
@@ -72,9 +73,15 @@ for package in $KEEL_PROJECT_PACKAGES; do
     echo "$package $installed, the candidate of the project archive"
 done
 
-# the build time package source is not for appliances: remove it, and keep the
-# documented, disabled entry for the future signed repository (overlay)
+# the build time package source is not for appliances: remove the source entry,
+# the copy of the archive it names and the keyring the build verified that
+# archive with, and keep the documented, disabled entry for the future signed
+# repository (overlay). The staging key signs whatever the build host produced,
+# so an image that kept it would carry trust in a nightly (tracker#7).
+# common/removelists-final/turnkey removes the same three paths at the end of
+# the build, whatever a recipe does.
 rm -f "$KEEL_STAGING_LIST"
+rm -f "$KEEL_STAGING_KEYRING"
 rm -rf "$KEEL_APT_ROOT"
 rm -rf "${KEEL_APT_LISTS:?}"/*
 grep -q '^Enabled: no' "$KEEL_SOURCES"