Skip to content

fix: images take packages from Debian and the Keel repository only - #30

Merged
marcos-mendez merged 5 commits into
19.xfrom
fix/keel-apt-sources
Oct 2, 2026
Merged

marcos-mendez merged 5 commits into
19.xfrom
fix/keel-apt-sources

Conversation

@marcos-mendez

@marcos-mendez marcos-mendez commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The maintainer found that the first-boot security-updates step of Keel images still fetches from archive.turnkeylinux.org. This is where that comes from. The step8 Web image ships these files, all of them written by conf/bootstrap_apt and overlays/bootstrap_apt:

  • security.sources.sources: a TurnKey stanza (trixie-security, tkl-archive-keyring.gpg, enabled) next to Debian security. This is the file 95secupdates and cron-apt read.
  • sources.sources: TurnKey trixie (enabled) next to Debian trixie.
  • turnkey-testing.sources (disabled), with TurnKey's keys trusted.
  • /etc/apt/preferences: o=turnkeylinux at 999.

After this change (handbook decisions 0039 and 0043, tracker#23):

file content
debian.sources deb.debian.org trixie trixie-updates, debian-archive-keyring.pgp
security.sources security.debian.org trixie-security, debian-archive-keyring.pgp
debian-backports.sources unchanged, disabled
keel.sources archive.keellinux.org trixie (stable track) enabled, trixie-testing disabled, keel-archive-keyring.gpg
preferences.d/keel Pin: release o=Keel Linux, Pin-Priority: 990
  • plans/turnkey/base installs keel-archive-keyring instead of turnkey-keys. Fingerprint AD0964BE3F09DED469A3B6B2148E951314703180.

  • Why 990. It is above Debian's 500, so a Keel package is the candidate whatever version Debian has. It is below 1000, so apt never installs it over a newer installed version. At 1001 it does (tracker#23). tests/apt-sources.bats asserts both against apt with local archives, and fails at 1001 and at 500 (checked by mutation).

  • conf/turnkey.d/keel-apt stops the build when:

    • keel.sources has no enabled https://archive.keellinux.org trixie stanza;
    • keel-archive-keyring is not installed, or is older than 0.1.1;
    • the pin on o=Keel Linux is missing, or anything but Pin-Priority: 990, in /etc/apt/preferences or any file of preferences.d. A recipe overlay's preferences.d/keel at 1001 is caught. The build's own o=Keel Linux Pool pin and the recipes' build-only l=Keel Linux staging pin are not matched; the final removelist takes both out.
    • anything still names TurnKey: /etc/apt/sources.list, sources.list.d, preferences, preferences.d, an unowned tkl-* keyring, or a key in trusted.gpg or trusted.gpg.d whose user id is TurnKey's (read with gpg, so a renamed file is caught).

    A turnkey-keys that a parent layer installed is purged with dpkg -P. Only the source files TurnKey's bootstrap wrote are removed by name.

  • bootstrap_apt removes the three names it used to write. Measured on the step8 image: left next to debian.sources, sources.sources makes apt refuse every source ("Conflicting values set for option Signed-By").

  • Goes with fix: first-boot security updates read Debian's security.sources inithooks#34 (95secupdates reads security.sources). Both must be in the same image build. An old inithooks with this common would upgrade nothing on first boot, because apt treats a missing sourcelist as empty.

The pool requirement (met, 2026-10-02)

Requirement. The captured pool that the next Core build is pointed at (keel-pool current) must hold keel-archive-keyring, Architecture all, at 0.1.1 or newer: conf/turnkey.d/keel-apt refuses an older one, since 0.1.0 shipped the revoked signing subkey. The package installs /usr/share/keyrings/keel-archive-keyring.gpg, primary key fingerprint AD0964BE3F09DED469A3B6B2148E951314703180. A pool without it fails the Core build at the plan (keel-archive-keyring not found), and every layer above Core inherits that.

What archive.keellinux.org serves now (read on 2026-10-02, after the release work published it):

suite package version .deb SHA256
trixie-testing keel-archive-keyring 0.2.0 927d49e88d2e1753d086bf4eb24abd7caedd6cc2ef0db470f5c697f550165399
trixie-testing turnkey-ssl 3.1.1+keel1 1fff5fd6503ed06b3beee86bc2cf9f64fe639328d87247851ae9ad4a3bd31995
trixie-testing turnkey-netinfo 1.1.1+keel2 d435cca4d8c061b02181f007cfc00a2750b8d641c390178f6a046359d3fa1f2b
trixie-testing turnkey-conffile 1.1.2+keel1 10b669a879000f0ba4af9cfd31389566324911be8557556a5fd4e7177740b0c5
trixie keel-archive-keyring 0.1.1 15ea1c3964da36d39808f64f42f6280c24ce3faedded9eb2f06489804545ff93

keel-archive-keyring.asc answers 200 at the archive root. The next Core build therefore captures keel-archive-keyring 0.2.0 from trixie-testing, and keel-apt accepts it (0.2.0 is above the 0.1.1 floor). The image's own keel.sources is unchanged by this: it still follows trixie, the stable track, with trixie-testing present but disabled. Which track a build reads is the release work's (decision 0039).

Earlier measurement on the build host, 2026-10-02, kept for the record:

  • The current pool, /srv/keel-pool/dists/2026-09-27, has 411 members captured from core. They include turnkey-keys 0.1 and TurnKey's own inithooks 2.3.6 and confconsole 2.2.3. No Keel package is in it, keel-archive-keyring included.
  • keel-pool capture fetches with the build host's own apt-get download --print-uris. The host's apt sources are deb.debian.org, security.debian.org and http://archive.turnkeylinux.org/debian. The Keel archive is not among them, so a capture cannot fetch keel-archive-keyring and would record it as a gap.
  • At that time only keel-archive-keyring 0.1.1 was published, in trixie and trixie-staging, and the build host's apt sources did not include the Keel archive, so a capture could not fetch it. Both are resolved by the publication above and the release work's build host setup.

The seven recipes, which merge first

Seven recipes shipped their own keel.sources (disabled) and preferences.d/keel (1001) at common's paths, and asserted Enabled: no. Those builds fail on this PR until these merge:

Each drops both overlay files. The 1001 pin had also been what made the build-time staging archive beat TurnKey's 999 pin during the recipe's upgrade, so each recipe now pins l=Keel Linux staging at 1001 for the build only (/etc/apt/preferences.d/keel-staging) and removes it with the staging source. This PR's final removelist removes it as well. keel-core and keel-web have neither file.

Merge order: the seven recipes, then the pool, then this PR together with inithooks#34.

TurnKey-built packages: no updates any more (review item 5)

archive.keellinux.org trixie has 6 packages. The step8 Web image has 36 packages whose name and version match TurnKey's trixie or trixie-security index exactly (checked against both indexes today). Without turnkey-keys, which this PR purges, that is 35:

  • webmin and 20 modules, all 2.660.turnkey0: authentic-theme, custom, fail2ban, fdisk, filemin, firewall, firewall6, logviewer, lvm, mount, net, passwd, postfix, raid, software, sshd, systemd, updown, useradmin, xterm. Also webmin-tklbam 1.2.1.
  • tklbam 1.5.3+2+g31134d3, tklbam-squid and tklbam-squid-common 2.7.STABLE9-2.3.0-2turnkey+0, turnkey-pypy2 7.3.23+turnkey1.
  • turnkey-conffile 1.1.2, turnkey-netinfo 1.1.1, turnkey-ssl 3.1.1, turnkey-sysinfo 1.1.0, turnkey-version 1.1.3, hubdns 1.4.0, py3curl-wrapper 2.1.2, tkl-dhcpcd-ifupdown-glue 0.1.3, tkl-installer 0.1.2.

Update, 2026-10-02: the first three are now Keel forks in trixie-testing (turnkey-ssl 3.1.1+keel1, turnkey-netinfo 1.1.1+keel2, turnkey-conffile 1.1.2+keel1; see the table above), so 32 remain TurnKey's builds. At 990 a +keelN fork is the candidate over TurnKey's version of the same package, which is exactly the case the pin exists for.

inithooks and confconsole are already Keel builds (+keelN).

Proposal (follow-up in Keel-Linux/apt): bin/keel-mirror-turnkey. It would publish those exact versions into Keel's trixie, so updates reach them through the Keel archive:

  1. Fetch them from archive.turnkeylinux.org trixie and trixie-security. Verify each .deb's SHA256 against TurnKey's signed Release, checked against TurnKey's key on the build host only and never on an appliance, plus its control fields. lib/pool.sh (pool_uris, pool_check) already does this per package.
  2. Include them into trixie with reprepro, signed with the Keel key. Each publication is one commit listing the packages.
  3. Run it on a schedule, and add a package only when its version moves.

In the long run each one is rebuilt from source as ours (tracker#15). turnkey-ssl is the first candidate, because it ships a private key (keel-core#8).

Other notes

Verification

The step8 Web image was patched in build order in an LXC container: bootstrap_apt, then keel-archive-keyring in place of turnkey-keys, then the overlay, then the conf scripts, then the new inithooks.

  • apt update contacted only deb.debian.org, security.debian.org and archive.keellinux.org. The Keel archive shows at 990 in apt-cache policy.
  • apt-get -s upgrade and apt-get -s dist-upgrade showed 0 downgrades. The image's inithooks 2.3.6+keel16, confconsole +keel11 and keel 0.15.1 are kept over the archive's keel5, keel2 and 0.3.5. With the pin set to 1001, dist-upgrade downgrades those 3 packages.
  • turnkey-install-security-updates: with libpng16-16t64 rolled back to the trixie version, the hook fetched 1.6.48-1+deb13u6 from security.debian.org.

Test plan

  • bats tests/apt-sources.bats tests/apt-identity.bats: 45 pass (the 990 pin check and the removelist RED first). The review fixes were written RED first, and replacing the purge with rm fails both purge tests.
  • shellcheck on bootstrap_apt, keel-apt, cronapt and the tests
  • kcov: conf/turnkey.d/keel-apt at 100 percent (44 of 44 lines)
  • CI green
  • the next Core/Web build reads the pool, and its image passes conf/turnkey.d/keel-apt

marcos-mendez pushed a commit that referenced this pull request Oct 2, 2026
Review of #30:

- turnkey-keys is purged with dpkg -P instead of its files being deleted
  by hand, so the package's files go with its dpkg record.
- The build fails when anything still names TurnKey: /etc/apt/sources.list,
  sources.list.d, preferences, preferences.d, a tkl-* keyring no package
  owns, or a key in trusted.gpg or trusted.gpg.d whose user id is
  TurnKey's (read with gpg, so a renamed file is caught).
- keel.sources must hold an enabled archive.keellinux.org trixie stanza:
  seven appliance recipes ship their own keel.sources, apt.keellinux.org
  and disabled, at the same path, and a recipe overlay wins.
- keel-archive-keyring must be installed and at least 0.1.1.
navigator added 5 commits October 2, 2026 17:30
The first boot's security updates fetched from archive.turnkeylinux.org.
conf/bootstrap_apt wrote an enabled TurnKey stanza into sources.sources
and into security.sources.sources (the file 95secupdates and cron-apt
read), turnkey-testing.sources beside them, TurnKey's keys imported and
an o=turnkeylinux pin at 999 from overlays/bootstrap_apt.

Per handbook decisions 0039 and 0043:

- conf/bootstrap_apt writes debian.sources (deb.debian.org trixie and
  trixie-updates), security.sources (security.debian.org trixie-security)
  and debian-backports.sources, signed by Debian's keyring, and removes
  the three names it used to write. Builds below Debian 13 are refused.
- overlays/turnkey.d/keel-apt adds keel.sources (archive.keellinux.org
  trixie, the stable track; trixie-testing disabled) signed by
  keel-archive-keyring, which plans/turnkey/base installs instead of
  turnkey-keys, and /etc/apt/preferences.d/keel at 990 on o=Keel Linux.
- 990 and not 1001 (tracker#23): above Debian's 500 the Keel version is
  the candidate whatever Debian publishes, below 1000 apt never installs
  it over a newer installed one.
- conf/turnkey.d/keel-apt stops a build missing keel.sources or the
  keyring, or with a source naming the TurnKey archive, and removes
  TurnKey's files a layer inherits by name, as apt-identity does.
- cron-apt's install actions read security.sources.

TurnKey's keys leave the image; the CI job that installs the overlay
packages keeps its trixie key as tests/fixtures/tkl-trixie-main.asc.
Review of #30:

- turnkey-keys is purged with dpkg -P instead of its files being deleted
  by hand, so the package's files go with its dpkg record.
- The build fails when anything still names TurnKey: /etc/apt/sources.list,
  sources.list.d, preferences, preferences.d, a tkl-* keyring no package
  owns, or a key in trusted.gpg or trusted.gpg.d whose user id is
  TurnKey's (read with gpg, so a renamed file is caught).
- keel.sources must hold an enabled archive.keellinux.org trixie stanza:
  seven appliance recipes ship their own keel.sources, apt.keellinux.org
  and disabled, at the same path, and a recipe overlay wins.
- keel-archive-keyring must be installed and at least 0.1.1.
Seven appliance recipes ship /etc/apt/preferences.d/keel at 1001 in
their overlay, which wins over common's 990 at the same path, and 1001
downgrades every package newer than the archive's (tracker#23). Every
stanza pinning o=Keel Linux in /etc/apt/preferences and preferences.d
must say 990, and there must be one. The build's own pool pin, o=Keel
Linux Pool at 1001, is not matched: removelists-final takes it out.
The seven recipes that read the staging archive now pin it at 1001 for
the build only, in /etc/apt/preferences.d/keel-staging, instead of
shipping /etc/apt/preferences.d/keel at 1001 in their overlay. The
removelist takes it out with the staging source, whether or not a recipe
remembers to.
@marcos-mendez
marcos-mendez merged commit a049bf6 into 19.x Oct 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant