Skip to content

image: systemd from Ubuntu's source package, stage1 and static - #271

Merged
aledbf merged 6 commits into
mainfrom
systemd-stage1
Nov 19, 2026
Merged

aledbf merged 6 commits into
mainfrom
systemd-stage1

Conversation

@aledbf

@aledbf aledbf commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The base image installs Ubuntu's own systemd, rebuilt here from its source package: the first SSH login is ~40-47 ms sooner, and the restore is unchanged.

What

image/systemd/Dockerfile builds systemd 259.9-0ubuntu0.1 from its .dsc, pinned by sha256 in versions.yaml (systemd-source). It keeps Ubuntu's patches and changes the build:

  • Debian's stage1 build profile: no BPF framework, audit, AppArmor, OpenSSL, TPM2, homed, cryptsetup, userdb…;
  • SELinux off;
  • static: systemd-executor (exec'd for every unit PID 1 starts), udevd, systemctl and journalctl link libsystemd-shared in.

The packages are +spin1 and held, so an apt upgrade in a workspace cannot bring back the archive's build. image/build.sh fails an image whose systemd is not this build, or whose packages are not held. The build itself asserts that meson took each option, that libsystemd-shared links no SELinux, AppArmor, audit or libcrypto, and that the executor loads no libsystemd.

The cost is that no unit can use RestrictFileSystems=, SocketBind*= or RestrictNetworkInterfaces=. None does, and spin's guard loads its own BPF.

Measured (bench fleet, 20 boots of each, interleaved)

first SSH login, p50 vs the archive's systemd run 1 run 2 run 3
stage1 −47 (44531842773) −38 (44537549305) −47 (44538877783)
static, over stage1 −10 −1 (accept → login −7 in both)

Also

  • boot:lockdown checks that an image with this systemd asks for nothing lockdown refuses: no bpf-restrict-fs, no "use of bpf to read kernel RAM". It reads systemctl --version and requires this boot's kernel start in the journal first, so an empty read cannot pass. Closes guest: bpf-restrict-fs fails to load on every boot #261.
  • boot:restore-first is the goal's second number as a probe; it has been missing since machine: every checkpoint is mapped-ram; boot:score holds a change to cold boot and restore #246 closed. It is in lab.yml.
  • task refresh:systemd, with its job in refresh.yml: the archive serves only the current source, so the build fails the day Ubuntu moves it, and this proposes the new pin that day.
  • SOURCES and the pins SBOM name the source and how it was built.
  • mkosi takes the packages as a local repository, which needs apt-utils in the builder. libpam-systemd is named in mkosi.conf: arriving through docker inside the chroot, it was the archive's, which wants the archive's systemd exactly.
  • docs/log.md: week 11, with these runs and the nested-against-metal numbers.

Checked

  • task image:build builds the image with all seven packages +spin1 and held: 217 packages, the same as before.
  • boot:lockdown (shows systemd 259 (259.9-0ubuntu0.1+spin1)), boot:logind and boot:hotadd pass on it.
  • task lint and go test ./... pass.

The fingerprint does not move: nothing in machine/ changes, and a resumed guest keeps the systemd it was saved with.

🤖 Generated with Claude Code

aledbf and others added 2 commits November 19, 2026 10:05
Ubuntu's own systemd 259.9-0ubuntu0.1, from its source package, rebuilt
with Debian's stage1 build profile and SELinux off: no audit,
cryptsetup, coredump, AppArmor, OpenSSL, TPM2, homed, userdb and no BPF
framework (socket-bind's ~25 ms in the verifier, restrict-fs's failed
load, #261). Versioned +spin1.

boot:ssh-first takes the packages as a row of their own beside the
image as built (SPIN_SYSTEMD_DEBS), put into the overlay the way dpkg
upgrades them, /etc left alone; lab.yml's systemd_run fetches them from
a Base image run.

A laptop, 20 boots of each interleaved, p50/p95 ms:
  as this branch  accepts 143/148  first login 237/245
  systemd stage1  accepts 119/127  first login 207/214

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
versions.yaml's systemd-source pins the .dsc by sha256, fetched from
the archive's deb-src. The image holds every +spinN package so an apt
upgrade cannot swap it for the archive's build, and build.sh checks
dpkg's status says hold. The upload is declared not published.
boot:waits takes SPIN_SYSTEMD_DEBS as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf

aledbf commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Bench fleet, lab run 44531842773, boot:ssh-first, v20261114.02, packages from run 44529972582. 20 boots of each, interleaved, 2 vCPUs, p50/p95 ms from the moment before QEMU is exec'd:

sshd accepts first login accept to login a second login console prompt
as released 243/259 390/414 145/155 49/53 435/459
as this branch 248/265 395/416 148/156 49/52 441/464
systemd stage1 215/230 348/375 133/145 45/52 387/418

The first login is 47 ms sooner at p50 and 41 ms at p95 (−12%), and no boot failed to log in. boot:waits on a laptop (10 boots of each) places the gain: PID 1's setup before the generators drops from 25.8 to 5.6 ms, which is the BPF framework the build leaves out.

Still to measure: the checkpoint restored to the same first login.

The goal's second number had no probe in the tree since #246 closed.
boot:restore-first boots each of boot:ssh-first's rows cold, logs in,
saves, then restores every state read from the disk and logs in again,
every cold boot before every restore as #246 found the nested fleet
needs. The rows are shared with boot:ssh-first (firstLoginRows), so
SPIN_SYSTEMD_DEBS adds the stage1 row to both. lab.yml offers it.

AGENTS.md: a socket's path fits in 108 bytes; the QMP sockets live in a
short directory of their own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf

aledbf commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

The second number: boot:restore-first, lab run 44533227886, bench fleet, 20 of each, every cold boot before every restore, each state evicted from the page cache. p50/p95 ms:

cold login save restore: running restore: login
as released 391/408 1205/1213 791/846 841/913
as this branch 393/412 1209/1213 790/838 840/890
systemd stage1 352/369 1104/1110 763/1255 815/1309

The restored guest's own part, running → login, is ~50 ms in every row, stage1's included. The p95 is three slow state reads (1.19-1.38 s to running, before the guest executes anything). as this branch had one of 2.0 s. That is the host's disk, not systemd. Stage1 does not make the restore worse.

Beside the question: reading the state is ~750 ms of the restore's ~820. That is much more than #246 measured on 11-13 (277 ms to running), and it is the restore's lever, not the guest.

image:systemd builds each variant into debs/<variant> with meson options
after the packaging's own, and checks meson took them. stage1-static links
systemd-executor (exec'd for every unit started), udevd, systemctl and
journalctl with libsystemd-shared instead of loading it. boot:ssh-first
and boot:restore-first make a row of each variant, and one of stage1 with
systemd-udev-load-credentials masked. No LTO variant: Ubuntu's build flags
already have -flto=auto.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf marked this pull request as ready for review November 19, 2026 13:47
@aledbf

aledbf commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Variants, lab 44537549305, bench fleet (nested, c8i.4xlarge), packages from run 44534853184, 20 boots of each interleaved, p50/p95 ms:

sshd accepts first login accept → login a second login
as this branch 237/256 378/407 143/153 46/54
stage1 207/228 340/375 132/145 45/52
stage1, no udev creds 205/234 332/373 129/140 46/49
stage1-static 204/219 330/349 125/130 43/46
  • stage1 holds: −38 ms p50, the same as the first run (−47).
  • Static linking (systemd-executor, udevd, systemctl and journalctl with libsystemd-shared linked in): another −10 ms p50 and −26 ms p95 over stage1. That fits removing a dynamic load from every unit's exec.
  • Masking systemd-udev-load-credentials: −8 ms p50, p95 unchanged. Within noise on its own.

Next row: static and the mask together.

Over stage1 alone the mask moved the first login -8 ms p50 and nothing
at p95 (lab 44537549305); it is asked again with the build that would
ship.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf

aledbf commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Static and the mask together, lab 44538877783, bench, 20 boots of each interleaved, p50/p95 ms:

sshd accepts first login accept → login console prompt
as this branch 242/250 383/395 143/146 427/440
stage1 206/214 336/349 132/137 379/390
stage1-static 209/220 335/351 125/130 371/385
stage1-static, no udev creds 210/218 329/341 120/126 367/377

Across the three runs (44531842773, 44537549305, this one):

  • stage1: −47, −38, −47 ms on the first login. It holds.
  • static over stage1: −10, then −1 ms on the first login, so within noise. Accept → login is −7 ms in both runs (132 → 125), and that is consistent. The packages are 1.7 MB larger and need nothing more to maintain.
  • the udev credentials mask: −8, then −6 ms. Small, and not separable from noise with 20 boots.

The base image installs Ubuntu's own systemd 259.9-0ubuntu0.1 rebuilt
by image/systemd/Dockerfile from its .dsc (versions.yaml systemd-source):
Debian's stage1 build profile, SELinux off, and systemd-executor, udevd,
systemctl and journalctl linked statically against libsystemd-shared.
The packages are versioned +spin1 and held; build.sh fails an image
whose systemd is not this build or whose packages are not held.

On the bench fleet, 20 boots of each interleaved, the first SSH login
p50 against the archive's systemd: stage1 -47, -38, -47 ms (labs
44531842773, 44537549305, 44538877783); static over stage1 -10 and -1,
accept-to-login -7 in both. The restored checkpoint's guest part is
unchanged (44533227886). PID 1 no longer loads socket-bind through the
BPF verifier (~20 ms of its setup) nor restrict-fs, which lockdown
refused at every boot: boot:lockdown holds that for an image built
with it (#261).

- task image:systemd, run by image:build, cached like the other builds;
- mkosi takes the packages as a local repository, which needs
  apt-utils in the builder, and libpam-systemd is named: arriving
  through docker's dbus-user-session inside the chroot it was the
  archive's, which wants the archive's systemd exactly;
- refresh:systemd and its job propose a new source the day Ubuntu
  publishes one, since the archive then stops serving the pinned one;
- SOURCES and the pins SBOM name the source and how it was built;
- the experiment's rows, lab input and CI job are removed;
  boot:restore-first stays.

Closes #261

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf aledbf changed the title experiment: systemd from Ubuntu's deb-src with the stage1 profile image: systemd from Ubuntu's source package, stage1 and static Nov 19, 2026
@aledbf
aledbf merged commit 08c09d3 into main Nov 19, 2026
7 checks passed
@aledbf
aledbf deleted the systemd-stage1 branch November 19, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

guest: bpf-restrict-fs fails to load on every boot

1 participant