TOOL-30719 build packages in a container bootstrapped for the target suite

[prakashsurya]

TOOL-30719

Why

Package builds run natively on the build host's root filesystem. setup.sh rewrites /etc/apt/sources.list to the target suite's mirror, and mk-build-deps --install then puts that suite's build dependencies onto the host itself. A build is therefore not isolated from the machine running it in either direction: it mutates that machine, and it reads whatever that machine already happens to have.

That couples the buildserver's OS to the OS being built, and the coupling bites hardest exactly when an Ubuntu LTS upgrade starts. The Delphix buildserver is itself an appliance image, assembled from the packages linux-pkg builds, so at the beginning of a cycle no machine exists that can build packages for the new suite: you need a buildserver on the new release to build the packages, and the buildserver is built from those packages. The standing workaround is to hand-bootstrap a throwaway one from a stock Ubuntu AMI of the new release (dc image import, clone a VM, run bootstrap/playbook.yml and expect install failures, expand root to 200G, add the docker group, snapshot with a long expiry) purely to get a single machine able to build once. Nothing exercises that path between cycles, so it is re-derived from scratch each time, and since an LTS upgrade comes around every year or two it is usually re-derived by someone who has not done it before.

The live instance is the 24.04 (noble) to 26.04 (resolute) cycle now in progress. What this change buys there is that the existing noble buildserver can build resolute packages directly, by pointing UBUNTU_DISTRIBUTION at the new suite, with no stock-AMI step at all. That variable is set per-branch in lib/common.sh and the os-upgrade branch is where a cycle bumps it, so the suite a container is bootstrapped for follows the branch being built rather than the machine doing the building. One prerequisite is unchanged and still ordered ahead of this: the new suite has to be published in the internal package mirror first, since that mirror is what debootstrap pulls the rootfs from.

The second half of the coupling — that a build silently reads from its host — is what the Jenkins validation below spent most of its time on. Four packages turned out to be taking build inputs from the appliance image without declaring them, and a clean root is what made that visible.

What changes

setup.sh now has two phases. On the host it bootstraps a root filesystem for $UBUNTU_DISTRIBUTION with debootstrap --variant=buildd and imports it as linux-pkg-build:$UBUNTU_DISTRIBUTION; inside the container it configures apt and installs the build tooling as before. Each entrypoint (buildpkg.sh, checkupdates.sh, sync-with-upstream.sh, push-merge.sh) re-execs itself into that image with the checkout bind-mounted at the same absolute path, so existing build code runs unchanged against a disposable root.

sbuild and pbuilder were considered and rejected. Their unit of work is a Debian source package built offline with dependencies declared in debian/control, whereas linux-pkg's unit is a config.sh with arbitrary hooks: virtualization runs ant and needs a docker daemon, zfs cross-builds modules against several kernel header trees fetched from S3, several packages declare build dependencies imperatively in prepare(), and builds need the network mid-build. Adopting either would mean re-expressing every package, which is a rewrite of the framework rather than a change to its build isolation.

Consequences worth knowing about:

• UBUNTU_DISTRIBUTION is environment-overridable rather than a hard literal. This is a deliberate behavior change beyond the original scope, and it is load-bearing: every cross-suite test depends on it, and while it was a literal any override was silently a no-op. Any environment exporting that name steers the suite, the image tag, and the apt sources.
• A host that is not an aws/internal-buildserver image now needs DISABLE_SYSTEM_CHECK=true, a plain Ubuntu AWS instance included. The host safety check asks the appliance image what it is, through get-appliance-platform and get-appliance-variant. Inferring this from AWS instance metadata cannot work, since metadata answers on any AWS instance, a developer's own cloud workstation included.
• In-container detection keys off a marker file baked into the image, not an environment variable passed to docker run. A variable is forgeable, and setting it on a host would skip the safety check and send setup.sh down the host path, rewriting that host's /etc/apt/sources.list and installing build packages onto it.
• Environment crossing the boundary is an explicit allowlist, so a variable the build needs and nobody listed fails loudly instead of silently changing what gets built.
• Packages that need a docker daemon declare PACKAGE_NEEDS_DOCKER and receive the host's socket, with --group-add derived from the socket's own gid, because docker does not propagate the host's supplementary groups. virtualization, docker-python-image and containerized-masking set it.
• The secondary mirror's key is scoped with signed-by= on its own source line. Installing it globally would give it authority over every source in the file, the primary Ubuntu mirror included; apt-key is also absent from 26.04. Verified against the live mirror: the key gives a good signature on the secondary ppas repo and cannot verify the primary ubuntu repo at all.
• The container carries a generated en_US.UTF-8 locale. The buildserver supplies one ambiently, via the locales package every appliance image installs, and a debootstrapped root has none. Generation and selection sit together in the Dockerfile so they cannot drift apart.
• zfs declares systemd as a build dependency. Its configure decides whether to install unit files by probing for the systemctl binary, which is present on any host running systemd and absent from a debootstrapped root. Without it configure resolves to --without systemd, the units are never generated, and dh_install aborts. That collision is the only reason this was loud: debian/*.install lists the unit paths unconditionally, so a package that globbed them would have shipped without its units instead of failing.
• setup.sh no longer arranges swap. It previously added a /swapfile, but returned early whenever the root filesystem was ZFS, which it always is on an appliance, so on every sanctioned build host it did nothing. delphix-platform provisions swap for the internal-buildserver variant. That provisioning was unreliable, which is what TOOL-30723 below was about; it is fixed on develop and waiting on a buildserver image roll.
• The image is rebuilt when the resolved mirror snapshot, the invoking user's uid or gid, or the Dockerfile's content changes. apt is repointed at the current snapshot on every run, so a stale rootfs would reintroduce the toolchain drift this removes; and the reuse check returns before docker build is reached, so docker's own layer cache never gets to notice a Dockerfile edit.
• Escape hatches: LINUX_PKG_NO_CONTAINER=true runs in place on the host, and ./buildpkg.sh -S <pkg> opens a shell with a build's mounts and environment for reproducing a failure.

The container's tool contract was discovered empirically rather than designed: wget, git, fakeroot, bc, sbsigntool, kmod, python3-pip and a pinned awscli go into setup.sh's in-container provisioning, each added in response to a specific observed failure and named in a comment next to it. gnupg is kept in the image despite having no known consumer, because confirming it is unnecessary needs a full build from an image built without it.

Testing

Where this stands

#	Item	Status
1	build-packages green for the appliance set	Green. All 36 requested packages built in-container in one run, plus combine-packages
2	combine-packages → appliance-build → image boots	In flight. The chain is proven end to end; the run using an entirely container-built set is building now
3	virtualization and zfs in-container with Jenkins credentials	Both green
4	sync-with-upstream.sh / push-merge.sh in-container	sync-with-upstream.sh green. push-merge.sh unreachable from this controller; gap documented below

Four container-attributable failures were found and fixed. Everything still red is pre-existing on develop or environmental, each with a control run to prove it.

Verified locally, before Jenkins

On a DCoA buildserver VM (noble, ZFS root, docker 29.7.1):

• A resolute (26.04) root filesystem bootstraps and imports on a noble host whose debootstrap has no resolute script until the symlink step creates one
• ptools, host-jdks and connstat build end to end in-container, connstat including cross-compiled and signed kernel modules for five flavors
• ptools builds end to end inside a resolute container on that noble host
• The host's dpkg -l is unchanged by a containerized build, which is the property the change exists for
• Artifacts land in workdir/artifacts owned by the invoking user, so the Publish stage can read them
• Exactly one docker run per invocation, i.e. the re-exec cannot recurse
• Exit codes propagate; a failed build leaves its container for inspection and prints the re-entry command
• LINUX_PKG_NO_CONTAINER=true still builds on the host path
• The docker socket is present with --group-add, and a container started that way reaches the daemon. Without --group-add the same container gets permission denied
• Cross-suite artifacts are attributable to the target suite, not the host: the resolute build's -dbgsym DWARF records rustc 1.93.1 where the host caps at 1.75.0. The libc6 floor was identical on both suites and is a dead end rather than evidence
• No control-field or shlibs differences between host-built and container-built for all three packages; container-vs-container builds are byte-identical
• make check clean

Verified on Jenkins

Run on the selfservice controller against this branch. Where a package failed, the same job was re-run against develop as a control, so that "the container caused this" is measured rather than inferred.

The headline run is build-packages/pre-push #7329: 31 of 31 child builds green, no failures, 6h24m. It requested the appliance set minus windows-connector plus upgrade-verify, so every package in the Delphix Appliance except windows-connector is container-built in that one run.

Path exercised	Run	Result
Orchestrated appliance set	build-packages/pre-push #7329	green, 6h24m, 31/31 children
Kernels + connstat + delphix-kernel + zfs	build-kernel/pre-push #2969, folded into #7329	green, 5h22m, all 8 packages
combine-packages	combine-packages/pre-push #6007, from #7329	green, 5.9 min
buildpkg.sh, hardest package	build-package/virtualization #2343	green, 54.9 min — docker socket, --group-add, Jenkins credentials, locale
buildpkg.sh, zfs standalone	build-package/zfs #3558	green, 157.7 min
sync-with-upstream.sh	update-package/drgn #3	green, real push to the dry-run staging ref, and its whole downstream chain green
combine-packages → appliance-build → boot → tests	appliance-build-orchestrator-pre-push #14722, from update-package/drgn #3	green, 6h32m
checkupdates.sh	check-updates/main #2	UNSTABLE on a 3h job timeout — pre-existing, see below
Appliance from an entirely container-built set	appliance-build-orchestrator-pre-push #14725, from #6007	in flight

Green in-container, all 36 requested packages: bcc, challenge-response, cloud-init, connstat, crash-python, crypt-blowfish, delphix-go, delphix-kernel, delphix-platform, delphix-rust, delphix-sso-app, docker-python-image, drgn, dwarves, fluentd-gems, gdb-python, grub2, host-jdks, libkdumpfile, linux-kernel-aws, linux-kernel-azure, linux-kernel-gcp, linux-kernel-generic, linux-kernel-oracle, makedumpfile, masking, misc-debs, nfs-utils, performance-diagnostics, ptools, python-rtslib-fb, savedump, sdb, targetcli-fb, virtualization, zfs. Plus upgrade-verify, which is not built by linux-pkg but is part of the combined set.

Item 2: how far the chain is proven

update-package/drgn #3 carried its own downstream all the way through, which is the first time anything past combine-packages has run on this branch:

sync-with-upstream.sh in-container            green
build-packages/pre-push/7326                  SUCCESS  28.4 min
combine-packages/pre-push/6000                SUCCESS
appliance-build-orchestrator-pre-push/14722   SUCCESS  6h32m
  appliance-build/develop/pre-push/7149          SUCCESS   image built
  delphix-build-and-snapshots/ami-snapshots/9997 SUCCESS   AMI snapshotted, so it booted
  dx-integration-tests/36205                     SUCCESS   tests on a booted engine

So combine-packages → appliance-build → boot → integration tests all work. What that run does not show is an appliance assembled from container-built packages: pre-push builds skip the S3 latest link update (UPDATE_LATEST_LINK, linux_pkg_build_package.groovy:249, logged as Skipping update of 'latest' link), so #7326's combined set is host-built latest artifacts plus container-built drgn alone, 1 of 37.

#14725 closes that gap. It is the same job with the same parameters, fed #7329's combine-packages output instead, where all 37 are container-built.

Failures, and who owns each

Package	Verdict	Owner
gdb-python	Latent undeclared dependency, surfaced by the clean root. Control on develop passes	Fixed here, green
docker-python-image	Needs a docker daemon. Control on develop fails earlier, for an unrelated pre-existing reason	Fixed here, green
virtualization	Container had no generated locale	Fixed here, green
zfs	Undeclared systemd build input, so configure silently disabled its unit files and dh_install aborted	Fixed here, green
containerized-masking	Two container gaps fixed here, in front of a pre-existing dms-core-gate defect. Control on develop also fails	Upstream; not in the appliance set
windows-connector	Pre-existing. Control on develop fails identically	dlpx-app-gate; not fixable from linux-pkg
check-updates 3h timeout	Pre-existing. Times out on develop too, nightly since 2026-07-28	TOOL-30727, fix in devops-gate#4682

zfs — undeclared systemd build input, found on the re-run

The earlier zfs failure was an OOM, which masked this one: the build died before it ever reached dh_install. With swap present it gets much further, through the libzpool link and the Rust object agent, and then:

dh_install: warning: zfsutils-linux missing files: lib/systemd/system/zfs-scrub@.service
dh_install: warning: zfs-zed missing files: lib/systemd/system/zfs-zed.service
dh_install: error: missing files, aborting
make[1]: *** [debian/rules:138: override_dh_install] Error 255

One line of configure output explains it, and it is the only signal given:

checking for systemd support... no

config/user-systemd.m4 leaves enable_systemd at check and resolves it by probing for the systemctl binary. A host running systemd always has one; a --variant=buildd debootstrap root has libsystemd0 and libudev1 but not systemd. The probe fails, configure resolves to --without systemd, the units are never generated, and debian/*.install lists them unconditionally, so dh_install aborts.

prepare() now installs systemd. configure reports checking for systemd support... yes, and every path from the error list is back in the package: zfs-zed.service, zfs-scrub@/zfs-trim@ and their weekly and monthly timers, zfs-volume-wait.service, zfs-volumes.target, zoa-chaos-monkey.service, lib/systemd/system-preset/50-zfs.preset, usr/lib/systemd/system-generators/zfs-mount-generator, and lib/modules-load.d/zfs.conf. Both zfsutils-linux and zfs-zed postinst carry their systemctl enable snippets, so the units are still enabled on install.

gdb-python — undeclared libkdumpfile build dependency

Broken gdb-python-build-deps:amd64 Depends on libkdumpfile:amd64 < none @un H >
  Removing gdb-python-build-deps:amd64 because I can't find libkdumpfile:amd64
mk-build-deps: Unable to install all build-dep packages

libkdumpfile is listed in the repo's debian/control Build-Depends, but it is built by linux-pkg rather than published in the Ubuntu archive, and config.sh declared no PACKAGE_DEPENDENCIES, so nothing fetched it. On the host it resolved silently against the copy the appliance image already carried. drgn has the same dependency and already handles it correctly, so gdb-python now does the same thing.

docker-python-image — needs the docker socket, and is already broken on develop

The control on develop fails earlier, for an unrelated reason: the buildserver carries docker-ce (docker-ce-cli 29.7.1, containerd.io), which Conflicts with docker.io/docker-cli/containerd, so mk-build-deps cannot install the docker.io its debian/control asks for at all. In a debootstrapped root there is no docker-ce, docker.io installs cleanly, and the build gets further before hitting the real missing input:

docker pull registry.delphix.com/python:2.7.18-slim
failed to connect to the docker API at unix:///var/run/docker.sock ... no such file or directory

debian/rules' override_dh_install needs a docker daemon, so the package now declares PACKAGE_NEEDS_DOCKER. The invoke-rc.d: unknown initscript, /etc/init.d/docker not found line in the log is a red herring: that is docker.io's postinst in a container with no init system, and prepare completes after it.

virtualization — the container had no generated locale

The socket and --group-add worked and prepare completed, but ant then ran for 33 minutes before javac failed on UTF-8 punctuation in source comments:

GetTimezone.java:23: error: unmappable character (0xE2) for encoding US-ASCII
9 errors
BUILD FAILED

packages/virtualization's build() already exports LANG=en_US.UTF-8, but exporting a locale is inert unless it has been generated, and a debootstrapped root has none: glibc falls back to C/POSIX and javac's default source encoding becomes US-ASCII.

This belongs to the environment rather than to any one package, and the evidence is that the host supplied it to every package equally — windows-connector never exports LANG, yet its javac compiles the same UTF-8 comments cleanly on the host and emits the diagnostic above in a container. A wrong locale is also silent rather than loud, changing encoding, sort order and date formatting instead of failing, which is the wrong thing to make each package remember to ask for. The Dockerfile installs locales, runs locale-gen en_US.UTF-8, and sets ENV LANG.

The fix is visible in #2343's log as locale-gen en_US.UTF-8... done and Step 6/12 : ENV LANG=en_US.UTF-8, with no unmappable character diagnostics and every ant/javac phase reporting BUILD SUCCESSFUL.

containerized-masking — two container gaps in front of an upstream defect

It has no prepare() stage, so no JDK is installed and the java-8 JAVA_HOME its build() exports points at nothing. Worth being precise about what this is not: masking sets the identical java-8 JAVA_HOME and builds fine, because dms-core-gate's own gradlew logs Setting JAVA_HOME to Java 17 for build and overrides it once it finds the openjdk-17-jdk that masking's prepare() installs. containerized-masking runs a different task in the same repo without installing that list, so the wrapper has nothing to switch to. It now installs the same list, and also declares PACKAGE_NEEDS_DOCKER, since it reaches :tools:docker:buildLocalDockerImage. Both stale JAVA_HOME lines are left alone, so neither package quietly changes which JDK builds the product.

Neither fix makes it green, because it fails on develop too. Both fixes are confirmed working in the latest run: openjdk-17-jdk installs, the wrapper logs Setting JAVA_HOME to Java 17 for build, and the build reaches and completes the docker image it was asked for. It then dies in dms-core-gate's tools/docker/build.gradle:98:

Successfully tagged delphix-masking-app:2026.5.0.0
> Task :tools:docker:buildLocalDockerImage FAILED
> Failed to query the value of task ... property 'imageId'.
   > Querying the mapped value of task ... property 'imageIdFile' before task ... has completed is not supported

The fixes are still worth having: without the socket the container failed earlier and misleadingly, reporting Connect to http://127.0.0.1:2375 failed: Connection refused rather than anything about docker being absent. This package is in neither build/main.pkgs nor build/kernel-modules.pkgs, so it gates nothing here.

windows-connector — pre-existing, not fixable from linux-pkg

Fails on both branches at the same place, appliance/host/windows/build.gradle:253 in :host:windows:buildDelphixConnectorService, which shells out to its own apt-get install for the mono toolchain and exits 100. Only apt's proximate complaint differs: on the control it is E: Error, pkgProblemResolver::Resolve generated breaks with python3 unmet; in-container it is E: Packages were downgraded and -y was used without --allow-downgrades on tzdata. That apt-get lives in dlpx-app-gate, not in a package config.sh. linux-pkg's own install_pkgs() does pass --allow-downgrades; this build bypasses that helper.

Because this failure stands, build-packages cannot report green with BUILD_POLICY=ALL, and its pre-push combine-packages stage runs after the build stage, so one failing package aborts the pipeline before any COMBINED_PACKAGES_S3_URL is produced. #7329 therefore requests the appliance set minus windows-connector, which is why it is 36 packages rather than 37.

check-updates — a 3h job timeout that develop hits too

check-updates/main #2 went UNSTABLE at the 3 hour mark. This is not a container problem and not a hang, both of which were earlier readings of it.

The nightly production job on the ops controller, host path, identical hardcoded timeout and identical package list, has timed out on every run since 2026-07-28:

#1220 UNSTABLE 2026-08-04  181.3 min      8 consecutive daily timeouts
...
#1213 UNSTABLE 2026-07-28  181.3 min
#1212 SUCCESS  2026-07-27  134.7 min      16 prior successes, median 119.4 min

Only the five linux-kernel-* packages cost anything; every other package finishes its stages in single-digit seconds. fetch_repo_from_git takes a deep path when DO_UPDATE_PACKAGE is true, fetching repo-HEAD and upstreams/develop in full, because detecting an upstream update needs history a --depth=1 clone does not have. Those fetches grow with the repositories:

              07-26   07-27   07-28   08-04
aws            1058    1170    1147    1277
azure          1236    1407    1595    1742
gcp            1116    1287    1278    1377
generic        2098    2586    killed  killed
oracle         1133    1256    never   never

generic is what breaks the run: 35 min on 07-26, 43 min on 07-27, and still fetching after 101.5 min on 08-04.

That the container is not implicated is measurable rather than assumed. The same repo and branch fetched by a build-package job, which uses --depth=1, takes 78 s in-container against 93 s on the host, i.e. the container is marginally faster. Comparing the same day's deep fetches, aws is 1253 s in-container against 1277 s on the host.

Filed as TOOL-30727 with a timeout bump in devops-gate#4682. The operational consequence is worth knowing independently of this PR: UPDATE_PACKAGES is true in production, so linux-kernel-generic and linux-kernel-oracle have had no upstream update detection for eight days, and UNSTABLE rather than FAILURE meant nothing flagged it.

One correction to an earlier claim in this PR: check-updates does not run the full setup.sh once per package. debootstrap and the image build each happen exactly once. What repeats is the container start plus the in-container apt install, 13 times at roughly a minute each, so baking the tool install into the image would recover about 13 minutes, not hours.

What the container costs in build latency

Measured across 19 packages with both a container-built and a host-built run. The overhead is close to constant per build rather than proportional to build size, so the ratio is large for trivial packages and small for the ones that set wall clock.

Host build size	n	Ratio	Median delta
under 10 min	13	2.00x	+4.3 min
10 to 40 min	3	1.15x	+4.8 min
over 40 min	3	1.11x	+4.5 min

Summed over the 19: 374 min host against 472 min container, 1.26x. Median absolute delta +4.5 min, range +0.5 to +18.1. masking is +0.5, virtualization +4.4, delphix-rust +3.8.

Of that, about 2.7 min is fixed setup, measured from the logs as debootstrap around 70 s, docker build around 40 s, and the in-container tooling install around 45 s. The remainder is each package's build dependencies, which the buildserver already had installed and a clean root must fetch. That part is not waste; it is the coupling this change removes.

It does not amortize in CI. Every build-package job runs dc clone-latest onto a fresh DCenter VM, so there is no local image cache to hit and debootstrap runs once per build. The reuse check helps repeated builds on one machine, i.e. a developer's, and never CI. Removing that cost in CI would mean publishing the image to a registry or baking it into the buildserver image.

Pipeline latency grows by less than 1.26x, because build-packages batches in parallel and its wall clock is set by the longest chain, build-kernel at 1.11x. #7329 took 6h24m, of which build-kernel was 5h22m.

grub2 is excluded from the table as non-comparable: its host runs predate DLPX-97018 grub2 package doesn't need to be rebuilt, so its apparent speedup is a package change rather than a container effect.

systemd units still ship, and are still enabled

zfs losing its units raised the obvious question of whether other packages lose theirs silently, since a package that globbed those paths would have shipped without them rather than failing. Audited by running dpkg-deb -c over the container-built artifacts of every package in the appliance set.

Only builds that probe the environment are exposed. zfs was the only one. nfs-utils passes --with-systemd explicitly, so there is nothing to detect; delphix-platform, app-gate and masking copy static unit files.

Package	Units in the container-built deb
delphix-platform	44	identical to the source-declared set
nfs-utils	17	nfs-server, nfs-mountd, nfs-idmapd, rpc-statd, nfs-client.target, proc-fs-nfsd.mount
virtualization	14	all 14 app-gate units, incl. delphix-mgmt, delphix-nginx, delphix-postgres@, delphix-stat
cloud-init	10
grub2, targetcli-fb, python-rtslib-fb, masking	2 each
performance-diagnostics, delphix-sso-app	1 each	delphix-telegraf.service, delphix-sso-app.service

Nineteen other packages ship no units at all, which is correct rather than a regression: savedump is a Python script, the rest are libraries and tools. Not audited: linux-kernel-*, which are kernel images, windows-connector, which does not build and ships no Linux units, and upgrade-verify, which linux-pkg does not build.

Enablement is structurally unaffected. dh_installsystemd generates the postinst and prerm snippets from the presence of .service files in debian/tmp, and the enabling itself runs deb-systemd-helper on the target at install time. Neither needs systemd in the build root.

Still to verify

• Item 2 with an entirely container-built set. #14725 is building from #7329's combine-packages output. The chain itself is already green through boot and integration tests via #14722.
• zfs reliability rather than zfs correctness. Its greens were all obtained on a buildserver image that predates the TOOL-30723 swap fix, so they represent the device-enumeration coin flip landing favourably, three times. The systemd fix is what the greens prove; robustness waits on the image roll.

Not proven

• Cross-suite verification used http://archive.ubuntu.com/ubuntu, because the Delphix resolute mirror does not exist yet. The code path is identical and only the URL differs, but the production mirror configuration for a new suite is unexercised, including the secondary PPA source.
• push-merge.sh never runs in-container. The update-package jobs on selfservice hardcode DRYRUN=true, and the pipeline only calls push-merge.sh when DRYRUN is false, which exists only on ops and pushes a real upstream merge onto a package's develop branch. Reaching it was judged not worth mutating a shared repo. The untested delta is narrow: the container boundary, the DRYRUN and PUSH_GIT_TOKEN allowlist entries, and push_to_remote against a real remote are all covered by the sync-with-upstream.sh run, leaving only git ls-remote and the final push target unexercised.
• Two of the deep kernel fetches under check-updates were about twice as slow in-container as the same day's production run (azure 3504 s against 1742 s, gcp 2595 s against 1377 s) while aws matched within 2%. The most likely explanation is contention, since the container run overlapped three concurrent build-packages runs while production runs at 03:06 on a quiet controller, but that is not demonstrated and a quiet-period control would settle it.

Consequently the LTS runbook's stock-AMI step is reframed as conditional rather than deleted, in both topics/ubuntu-lts-upgrade-runbook.md and the 26.04 project page.

Related

• TOOL-30727 — check-updates has timed out nightly since 2026-07-28, so linux-kernel-generic and linux-kernel-oracle upstream updates are silently skipped. Pre-existing and reproducing on develop; surfaced by this work. Timeout bump in devops-gate#4682.
• TOOL-30723 — buildserver swap task hardcoded /dev/nvme1n1, which is sometimes the root pool disk, so a build VM could come up with no swap. Filed from this work, independent of it. Fixed on develop by delphix-platform#566, waiting on a buildserver image roll.
• DLPX-98274 — sign_modules() swallows a failed kmodsign because its loop runs in a subshell, so a signing failure would ship an unsigned module while reporting success. Pre-existing, deliberately not fixed here.

🤖 Generated with Claude Code

[lyriclake]

nice!

[lyriclake]

[nit] perhaps could shorten this to "The suite package builds run against, which may be different than the branch's default (see lib/container.sh)."

[lyriclake]

just for my understanding, what is this "9999.0.0.0" ? some special version value we use for something?

[prakashsurya]

it comes from here: https://github.com/delphix/linux-pkg/blob/develop/setup.sh#L36 .. I'm not introducing it, just moving the location..

doing a bit of digging, I think it originates from #338 and even before that, but we might have lost that history, if it was on a project branch..

it looks like when palash was previously doing the os-upgrade work, he set the product version to 999.0.0.0 since we didn't know what actual delphix version the new LTS would land on.. and in context of that new LTS, we were using a new package mirror host.. so we needed some way for new LTS builds to use the new package mirror host, and he settled on using a "fictional" delphix version, that was practically not going to collide with any real delphix version (we were still making actual delphis releases and version bumps, while that os-upgrade project branch was being worked on)..

we can likely revisit this decision, and probably remove this, since currently, we'll use the same package mirror host for develop, as well as the next LTS work for 26.04..

for now, I'll leave this as is, but open a follow up JIRA ticket to track the removal of this cruft.. I'm also unsure off the top of my head, what delphix version the os-upgrade branch will have for the 26.04 work.. I'm guessing it's still set to 9999.0.0.0, but I'll need to double check..

[prakashsurya]

yes.. looking at branch_schemas.toml in devops-gate, os-upgrade is set to 9999.0.0.0..

   34 [[branch_schemas.branches]]
   35 name = "os-upgrade"
   36 active = true
   37 version = "9999.0.0.0"

that's where this version value is coming from.. i.e. it has a backed in assumption about what the delphix version will be for the os-upgrade branch.. which is not obvious or intuitive when looking at this logic in isolation..

[lyriclake]

should this Task 1 ref be removed? ie if it's a reference to the implementation/test process.

[prakashsurya]

yes, thanks.. I'll do that.

[dbjwhs-perforce]

LGTM