Skip to content

Make a build that installs yesterday's packages fail - #7

Merged
marcos-mendez merged 1 commit into
mainfrom
fix/archive-freshness
Sep 27, 2026
Merged

marcos-mendez merged 1 commit into
mainfrom
fix/archive-freshness

Conversation

@marcos-mendez

Copy link
Copy Markdown
Contributor

The forum layer rebuilt on 2026-09-26 carried inithooks 2.3.6+keel1,
confconsole 2.2.3+keel1 and keel 0.1.0 after 2.3.6+keel4, 2.2.3+keel2 and
0.2.1 had been published to the staging archive. Every step of the build
succeeded and nothing in the recipe objected.

Why it passed

The archive the chroot read was the copy a build from that morning had left
in the bootstrap. fab stamps the bootstrap target, make clean failed on a
busy deck (rmdir: failed to remove build/root.patched: Device or resource busy), the stamps survived and the copy with them, so only root.patched
was rebuilt and apt-get install --only-upgrade found nothing newer than
what was already installed.

The recipe's own guard was grep -c '+keel1$', a literal that the stale
packages satisfied exactly.

What this changes

bin/keel-archive-check compares the package index copied into a build tree
with the index of the live archive and stops the build when they differ. The
Makefile calls it in bootstrap/post, right after the copy is made, and
again in root.patched/pre, which runs on every build and so catches a
bootstrap that this build did not make.

conf.d/zz-project-packages replaces the literal. For each of inithooks,
confconsole and keel it asserts that

  1. the archive offers exactly one version of it,
  2. apt's candidate is that version,
  3. that version is served by the project archive and not by an upstream
    source, so the package is a project build, and
  4. the installed version is that candidate, with dpkg status
    install ok installed.

It then removes the build time package source, which conf.d/main used to
do. No version is written down anywhere, so a rebuild made after a
publication either carries the newly published versions or fails.

Tests

Both scripts run for real against scratch trees, with dpkg, dpkg-query
and apt-cache as PATH stubs driven by fixtures: 21 new tests, no chroot, no
apt, no root. Coverage of the measured files, threshold 95:

 100.00  29/29  zz-project-packages
 100.00  43/43  nodebb.sh
  96.97  32/33  40nodebb
 100.00  124/124  boot-test-lib.sh
 100.00  26/26  keel-archive-check

Test plan:

  • tests/coverage.sh locally, 107 tests, every measured file at or above
    the threshold
  • shellcheck -S warning clean on both new scripts
  • the Makefile hooks expand as expected under fab's stamped target
    template (bootstrap/post after the copy, root.patched/pre before
    the conf scripts)
  • rebuild of the three layers on the build host, audited for
    inithooks 2.3.6+keel4, confconsole 2.2.3+keel2 and keel 0.2.1
  • appliance / build-and-boot on the rebuilt, published layer

The forum layer rebuilt on 2026-09-26 carried inithooks 2.3.6+keel1,
confconsole 2.2.3+keel1 and keel 0.1.0 after 2.3.6+keel4, 2.2.3+keel2 and
0.2.1 had been published, and every step of the build succeeded.

Two causes, both fixed here. The archive the chroot read was the copy a
build from that morning had left in the bootstrap: fab stamps the bootstrap
target, "make clean" failed on a busy deck, the stamps survived and the
copy with them. And the recipe asserted a literal "+keel1", which the stale
packages satisfied.

bin/keel-archive-check compares the package index copied into a build tree
with the index of the live archive and stops the build when they differ. The
Makefile runs it in bootstrap/post, right after the copy is made, and again
in root.patched/pre, which runs on every build and so catches a bootstrap
this build did not make.

conf.d/zz-project-packages replaces the literal. For inithooks, confconsole
and keel it checks that the archive offers exactly one version, that apt's
candidate is that version, that the version is served by the project archive
and not by an upstream source, and that the installed package is that
candidate and is configured. It then removes the build time package source,
which conf.d/main used to do. Nothing names a version, so a rebuild after a
publication either carries the new versions or fails.

Both scripts run for real in the tests against scratch trees with dpkg,
dpkg-query and apt-cache as PATH stubs: 21 tests, 100 percent of both files
under kcov, and they join the measured set in tests/coverage.sh.
@marcos-mendez
marcos-mendez merged commit ffb23e2 into main Sep 27, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant