diff --git a/.gitignore b/.gitignore index 10e54b0b..817249b6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ tmp/ /artifacts/ .idea +.superpowers/ diff --git a/README.md b/README.md index 4a5aa0b4..8c321429 100644 --- a/README.md +++ b/README.md @@ -29,26 +29,79 @@ projects. ## System Requirements -This framework is intended to be run on an Ubuntu 24.04 system with some basic -developer packages installed, such as git, and passwordless sudo enabled. Note -that it will automatically install various build-dependencies on the system, so -as a safety precaution it is currently restricted to only run on an AWS instance -to prevent developers accidentally running it on their personal machines. To -bypass the safety check, you can run the following command before running any -script: +Package builds run inside a Docker container image tagged +`linux-pkg-build:$UBUNTU_DISTRIBUTION`. That image's root filesystem is +bootstrapped with `debootstrap --variant=buildd` from the Ubuntu suite named by +`UBUNTU_DISTRIBUTION` (see [Environment Variables](#environment-variables)), so +a build runs against a rootfs for the suite being targeted rather than against +whatever happens to be installed on the host. Because of that, the host's own +Ubuntu codename no longer needs to match the suite being built; a `noble` host +can build packages for `resolute`, or any other suite the package mirror +serves. + +The host itself needs: + +* a reachable docker daemon (`docker info` must succeed, and your user must be + able to talk to it, typically by being in the `docker` group) +* `debootstrap` installed +* passwordless sudo, since bootstrapping and importing the container's root + filesystem requires root + +Note also that kernel headers and debug symbols for every flavor in +`TARGET_KERNEL_FLAVORS` now land under the docker root directory rather than +directly in the root filesystem. On a buildserver that is the same ZFS pool, so +there is no new capacity requirement, but a full disk surfaces as a docker layer +write failure rather than as an out-of-space error from the build. + +Every host-side invocation of any script, not just the one that happens to +build the container image, checks that the host is a Delphix buildserver image +before doing anything else, and refuses to proceed otherwise. It asks the image +what it is, via `get-appliance-platform` and `get-appliance-variant`, and +requires `aws` and `internal-buildserver` respectively. The check runs every +time because the host phase mutates the host itself whenever it does run, for +instance by teaching `debootstrap` about a new suite and creating +`/var/lib/linux-pkg`, and it exists as a safety precaution to keep developers +from accidentally running it on their personal machines. To bypass it, run +the following command before running any script: ``` export DISABLE_SYSTEM_CHECK=true ``` +On any host that is not a buildserver image that has to be set for every +invocation, not just the first. + +Because the build runs in a container, only what is passed explicitly reaches +it. Environment variables cross through an allowlist in `lib/container.sh` +(`CONTAINER_ENV_ALLOWLIST`, plus the `AWS_*`, `SECRET_DB_*`, `*_S3_URL` and +per-package `_GIT_URL` / `_GIT_BRANCH` / `_REVISION` patterns). +Files do not cross at all unless they are bind-mounted, and four are: the +checkout itself, `~/.aws` when it exists (read-only, so an AWS profile or +credentials file keeps working alongside credentials passed in `AWS_*` +variables), the key named by `SECRET_DB_JUMP_BOX_PRIVATE_KEY`, and the host's +docker socket for a package that declares `PACKAGE_NEEDS_DOCKER`. + +If you need to skip the container entirely, for example to debug the container +mechanism itself, set `LINUX_PKG_NO_CONTAINER=true` and the build will run in +place on the host using whatever is already installed there. This is an escape +hatch rather than the normal path, and the buildserver-image safety check above +still applies to it, on every invocation. So does the check that the host's own +Ubuntu codename matches `UBUNTU_DISTRIBUTION`: without a container, `setup.sh` +rewrites this host's `/etc/apt/sources.list` and installs from it, so the host +has to be running the suite being built. + ## Getting Started This quick tutorial shows how to build the packages managed by this framework. ### Step 1. Create build VM -You need a system that meets the requirements above. For Delphix developers, you -should clone the `dlpx-internal-buildserver-develop` group on DCoA. +Clone the `dlpx-internal-buildserver-develop` group on DCoA. The safety check +described above requires a buildserver image, so that is the supported path. +Which Ubuntu release the buildserver itself runs no longer has to match the +suite being built, since builds run against a rootfs bootstrapped for the target +suite rather than against the host's own packages. Any other host needs +`DISABLE_SYSTEM_CHECK=true` on every invocation. ### Step 2. Clone this repository @@ -165,6 +218,19 @@ in the `artifacts` sub-directory. Note that if the build of the package depends on build artifacts from another linux-pkg package, those will be fetched from a predetermined S3 location. +Pass `-S` instead of building to open an interactive shell in a build +container, with the same mounts and environment a build of that package would +get: + +``` +./buildpkg.sh -S +``` + +This is useful for reproducing a build environment or debugging a failure by +hand. Note that this shell does not run `setup.sh`, so it starts without the +build tooling `setup.sh` installs; run `setup.sh` yourself inside the shell +first if you need it. + ### checkupdates.sh Usage: @@ -229,9 +295,58 @@ to run this script. However, the script will fail unless DRYRUN is set to There's a set of environment variables that can be set to modify the operation of some of the scripts defined above. -* **DISABLE_SYSTEM_CHECK**: Set to "true" to disable the check that makes sure - we are running on the appropriate Ubuntu distribution in AWS. - Affects all scripts. +* **DISABLE_SYSTEM_CHECK**: Set to "true" to disable the safety check that + restricts every host-side invocation of any script to Delphix buildserver + images (`get-appliance-platform` reporting `aws` and `get-appliance-variant` + reporting `internal-buildserver`). That check runs on every invocation, not + just the one that happens to build the container image, so on any other host + this needs to be set every time, not just once. That includes a plain Ubuntu + AWS instance, which is not a buildserver image and so does not satisfy the + check on its own. Affects all scripts. This variable is read on the host + only; it is not passed into the build container, and so has no effect on the + codename check made there, which compares the container's rootfs against + `UBUNTU_DISTRIBUTION` and holds by construction. On the + `LINUX_PKG_NO_CONTAINER` path that same codename check runs on the host + instead, where it is the only thing verifying that the host runs the suite + whose apt sources `setup.sh` is about to write, and there this variable does + disable it. Use it with care on that path. + +* **UBUNTU_DISTRIBUTION**: The Ubuntu suite (codename) that packages are built + for, such as "noble" or "resolute". Defaults to the value hardcoded for the + current branch in `lib/common.sh`, but can be overridden in the environment + to build for a different suite than the branch's default. Determines the + tag of the build container image (`linux-pkg-build:$UBUNTU_DISTRIBUTION`) + and the suite `debootstrap` bootstraps that image's rootfs from. + +* **LINUX_PKG_NO_CONTAINER**: Set to "true" to run in place on the host + instead of inside the build container. This is an escape hatch for when the + container mechanism itself needs to be bypassed or debugged; the + buildserver-image safety check above still applies to it. Affects every script + that re-execs itself inside the container via `container_reexec()` + ([buildpkg.sh](#buildpkgsh), [checkupdates.sh](#checkupdatessh), + [sync-with-upstream.sh](#sync-with-upstreamsh), + [push-merge.sh](#push-mergesh)), and [setup.sh](#setupsh), which checks it + directly. + +* **LINUX_PKG_REBUILD_IMAGE**: Set to "true" to force the build container + image to be rebuilt from a freshly bootstrapped rootfs, even if an image + tagged `linux-pkg-build:$UBUNTU_DISTRIBUTION` already exists. Setting this is + not needed for the three cases where a reused image would be wrong, since + those rebuild it on their own: the image records the mirror snapshot its + rootfs was bootstrapped from, the uid/gid it was built for, and a hash of the + `Dockerfile` that produced it, and a build whose resolved snapshot, invoking + user or `Dockerfile` differs rebuilds rather than reuses it. Otherwise a + long-lived buildserver would keep compiling against a `debootstrap` toolchain + frozen at the image's build time while apt is repointed at the current + snapshot on every build, a second user on a shared host would run as a uid + the image has no `/etc/passwd` entry for, and an edit to the `Dockerfile` + would never reach an existing image, since this check returns before + `docker build` is reached and docker's own layer cache never gets a say. + +* **LINUX_PKG_BUILD_ROOT_DIR**: Directory where the build container's rootfs + is bootstrapped before being imported into docker. Defaults to + `/var/lib/linux-pkg`. Must point at a filesystem that is not mounted + `nodev` or `noexec`, since `debootstrap` refuses to install into either. * **DRYRUN**: Must be set to either "true" of "false" when running script [sync-with-upstream.sh](#sync-with-upstreamsh), and to "false" when running diff --git a/buildpkg.sh b/buildpkg.sh index 3ff90fe8..891afb2a 100755 --- a/buildpkg.sh +++ b/buildpkg.sh @@ -17,9 +17,7 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" - -logmust check_running_system -logmust run_setup_if_needed +ORIGINAL_ARGS=("$@") function usage() { [[ $# != 0 ]] && echo "$(basename "$0"): $*" @@ -40,6 +38,7 @@ function usage() { echo " -r override default revision for package." echo " -h display this message and exit." echo " -l use locally-built dependencies instead of s3 versions." + echo " -S start an interactive shell in the build container." echo "" exit 2 } @@ -49,8 +48,9 @@ unset PARAM_PACKAGE_GIT_BRANCH unset PARAM_PACKAGE_REVISION do_checkstyle=false +open_shell=false source="s3" -while getopts ':b:cg:hlr:' c; do +while getopts ':b:cg:hlr:S' c; do case "$c" in g) export PARAM_PACKAGE_GIT_URL="$OPTARG" ;; b) export PARAM_PACKAGE_GIT_BRANCH="$OPTARG" ;; @@ -58,6 +58,7 @@ while getopts ':b:cg:hlr:' c; do c) do_checkstyle=true ;; h) usage >&2 ;; l) source="local" ;; + S) open_shell=true ;; *) usage "illegal option -- $OPTARG" >&2 ;; esac done @@ -66,7 +67,38 @@ shift $((OPTIND - 1)) [[ $# -gt 1 ]] && usage "too many arguments" >&2 PACKAGE=$1 +# +# Checked here rather than after the re-exec so that a mistyped package name +# fails immediately, instead of after bootstrapping a container image and +# provisioning it, which is minutes of work for a name that was never going to +# resolve. Checked again inside the container by load_package_config(). +# logmust check_package_exists "$PACKAGE" + +# +# Whether this package's build needs the host's docker socket bound in has to be +# known before the container is started; see package_needs_docker(). +# +logmust package_needs_docker "$PACKAGE" +export PACKAGE_NEEDS_DOCKER="$_RET" + +if $open_shell; then + logmust load_package_config "$PACKAGE" + logmust container_shell +fi + +# +# Deliberately not logmust: this script's arguments can carry a credential +# ('-g https://@github.com/...'), and logmust would echo them verbatim +# into the build log. container_reexec() logs its own command line with any such +# URL masked, the way git_fetch_helper() and push_to_remote() do (lib/common.sh), +# and it exits with the container's status rather than returning a failure for +# logmust to catch. +# +container_reexec "./$(basename "$0")" "${ORIGINAL_ARGS[@]}" +logmust check_running_system +logmust run_setup_if_needed + check_env DEFAULT_GIT_BRANCH # diff --git a/checkupdates.sh b/checkupdates.sh index 51ee96c8..868a30f3 100755 --- a/checkupdates.sh +++ b/checkupdates.sh @@ -17,9 +17,7 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" - -logmust check_running_system -logmust run_setup_if_needed +ORIGINAL_ARGS=("$@") function usage() { [[ $# != 0 ]] && echo "$(basename "$0"): $*" @@ -47,6 +45,10 @@ shift $((OPTIND - 1)) [[ $# -gt 1 ]] && usage "too many arguments" >&2 PACKAGE=$1 +logmust container_reexec "./$(basename "$0")" "${ORIGINAL_ARGS[@]}" +logmust check_running_system +logmust run_setup_if_needed + logmust check_package_exists "$PACKAGE" # diff --git a/lib/common.sh b/lib/common.sh index e88a9bcd..a179d9c6 100644 --- a/lib/common.sh +++ b/lib/common.sh @@ -17,6 +17,8 @@ export _RET export _RET_LIST +export _RET_MIRROR_MAIN +export _RET_MIRROR_SECONDARY export _SECRET_BUILD_ARGS export DEBIAN_FRONTEND=noninteractive @@ -28,7 +30,13 @@ export SUPPORTED_KERNEL_FLAVORS="generic aws gcp azure oracle" # export JENKINS_OPS_DIR="${JENKINS_OPS_DIR:-jenkins-ops}" -export UBUNTU_DISTRIBUTION="noble" +# +# The suite package builds run against, which may be different than the +# branch's default (see lib/container.sh). +# +export UBUNTU_DISTRIBUTION="${UBUNTU_DISTRIBUTION:-noble}" + +source "$(dirname "${BASH_SOURCE[0]}")/container.sh" # # We currently support getting the linux kernel from 3 different sources: @@ -95,36 +103,43 @@ function logmust() { } # -# Check that we are running in AWS on an Ubuntu system of the appropriate -# distribution. This is not a strict requirement for the build to work but -# rather a safety measure to prevent developers from accidentally running the -# scripts on their work system and changing its configuration. +# Package builds install build dependencies and kernel headers into the root +# filesystem they run on, so they must run inside the build container, whose +# root is disposable and whose codename matches the suite being built. The +# codename comparison below therefore holds by construction, and exists to catch +# a mismatched or stale image rather than to guard the caller's machine. +# +# On the LINUX_PKG_NO_CONTAINER escape hatch there is no container, and the same +# comparison becomes the only thing verifying that this host runs the suite whose +# apt sources setup.sh is about to point it at, and whose packages it is about to +# install. The two cases need different advice when they fail, so the failure +# below distinguishes them. # function check_running_system() { - local msg - if [[ "$DISABLE_SYSTEM_CHECK" == "true" ]]; then echo "WARNING: System check disabled." return 0 fi - msg="Note that you can bypass this check by setting environment" - msg="${msg} variable DISABLE_SYSTEM_CHECK=true. Use this at your" - msg="${msg} own risk as running this command may modify your system." + if ! running_in_container && [[ "$LINUX_PKG_NO_CONTAINER" != "true" ]]; then + die "Expected to be running inside the build container." \ + "This is a bug in the re-exec logic in lib/container.sh." + fi if ! (command -v lsb_release >/dev/null && [[ $(lsb_release -cs) == "$UBUNTU_DISTRIBUTION" ]]); then - echo_error "Script can only be run on an ubuntu-${UBUNTU_DISTRIBUTION} system." - echo_bold "$msg" - exit 1 - fi - - if ! curl "http://169.254.169.254/latest/meta-datas" \ - >/dev/null 2>&1; then - echo_error "Not running in AWS, are you sure you are on the" \ - "right system?" - echo_bold "$msg" - exit 1 + echo_error "Running on ubuntu-$(lsb_release -cs 2>/dev/null || echo unknown)" \ + "but UBUNTU_DISTRIBUTION is '$UBUNTU_DISTRIBUTION'." + if running_in_container; then + die "The build container's rootfs does not match the suite being" \ + "built. Rebuild the image with LINUX_PKG_REBUILD_IMAGE=true." + else + die "LINUX_PKG_NO_CONTAINER is set, so this runs directly on this" \ + "host, which must itself run the suite being built. Either" \ + "unset LINUX_PKG_NO_CONTAINER and let the build run in a" \ + "container bootstrapped for $UBUNTU_DISTRIBUTION, or set" \ + "UBUNTU_DISTRIBUTION to this host's own suite." + fi fi } @@ -145,6 +160,45 @@ function run_setup_if_needed() { echo_bold "------------------------------------------------------------" } +# +# Resolve the URLs of the primary (Ubuntu archive) and secondary (PPA) package +# mirrors for the current branch, without modifying the system. If the URLs were +# passed in via the environment they are used as-is; otherwise the latest mirror +# snapshot for the branch is looked up. +# +# Sets _RET_MIRROR_MAIN and _RET_MIRROR_SECONDARY. +# +function resolve_mirror_urls() { + check_env DEFAULT_GIT_BRANCH DELPHIX_RELEASE_VERSION + local package_mirror_url latest_url delphix_version + local primary_url="$DELPHIX_PACKAGE_MIRROR_MAIN" + local secondary_url="$DELPHIX_PACKAGE_MIRROR_SECONDARY" + + if [[ -z "$primary_url" ]] || [[ -z "$secondary_url" ]]; then + delphix_version="$DELPHIX_RELEASE_VERSION" + if compare_versions "$delphix_version" eq "9999.0.0.0" || + compare_versions "$delphix_version" gt "2025.3"; then + latest_url="http://linux-package-mirror-v2.delphix.com/" + else + latest_url="http://linux-package-mirror.delphix.com/" + fi + + if is_release_branch; then + package_mirror_url="${latest_url}releases/${DELPHIX_RELEASE_VERSION}" + else + latest_url+="${DEFAULT_GIT_BRANCH}/latest/" + package_mirror_url=$(curl -LfSs -o /dev/null -w '%{url_effective}' \ + "$latest_url" || die "Could not curl $latest_url") + package_mirror_url="${package_mirror_url%/}" + fi + [[ -z "$primary_url" ]] && primary_url="${package_mirror_url}/ubuntu" + [[ -z "$secondary_url" ]] && secondary_url="${package_mirror_url}/ppas" + fi + + _RET_MIRROR_MAIN="$primary_url" + _RET_MIRROR_SECONDARY="$secondary_url" +} + function is_release_branch() { check_env DEFAULT_GIT_BRANCH [[ "$DEFAULT_GIT_BRANCH" == release/* ]] @@ -238,6 +292,7 @@ function reset_package_config_variables() { PACKAGE_PREFIX FORCE_PUSH_ON_UPDATE SKIP_COPYRIGHTS_CHECK + PACKAGE_NEEDS_DOCKER " for var in $vars; do diff --git a/lib/container.sh b/lib/container.sh new file mode 100644 index 00000000..e180c597 --- /dev/null +++ b/lib/container.sh @@ -0,0 +1,565 @@ +#!/usr/bin/env bash +# +# Copyright 2026 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Package builds run inside a container whose root filesystem is bootstrapped +# from the Ubuntu suite named by UBUNTU_DISTRIBUTION. This decouples the suite +# being built from the suite the build host runs, which is what allows a +# buildserver running one LTS release to build packages for the next one. +# + +# +# Home directory of the user that resources/Dockerfile.build-container creates, +# which is what $HOME resolves to inside the build container. The host user's +# home is a different path and is not mounted, so anything that has to be +# reachable through '~' inside the container has to land here. +# +CONTAINER_HOME="/home/delphix" + +# +# Name of the image used to build packages for the current suite. +# Sets _RET. +# +function container_image_tag() { + check_env UBUNTU_DISTRIBUTION + _RET="linux-pkg-build:${UBUNTU_DISTRIBUTION}" +} + +# +# File that resources/Dockerfile.build-container bakes into the build image, and +# whose presence is what identifies a process as running inside that container. +# +# The image is asked rather than the environment because this answer decides real +# work, not just cosmetics: setup.sh reads it to choose between preparing a +# container image and configuring the system it is running on directly. An +# environment variable would let anything on the host claim to be in a container +# just by setting it, and a host that made that claim would get its own +# /etc/apt/sources.list rewritten and build packages installed onto it -- exactly +# what check_host_is_disposable() below exists to prevent. A file baked into the +# image cannot be produced by setting a variable, and forging it needs root on +# the host. +# +CONTAINER_MARKER_FILE="/etc/linux-pkg-build-container" + +# +# True when the current process is already running inside the build container. +# +function running_in_container() { + [[ -f "$CONTAINER_MARKER_FILE" ]] +} + +# +# Refuse to treat this host as disposable unless it is a Delphix buildserver +# image, which is what every real build host is. Guards every path that is about +# to bootstrap a container image or, on the LINUX_PKG_NO_CONTAINER escape hatch, +# run the equivalent setup directly: both do the same irreversible things to +# whatever host runs them (sudo ln -s into /usr/share/debootstrap/scripts, sudo +# mkdir -p under /var/lib, debootstrap, sudo rm -rf, and on the escape hatch, +# rewriting /etc/apt/sources.list and installing packages). +# +# A Delphix image can state what it is, so it is asked directly. Inferring this +# from AWS instance metadata would not work: metadata answers on every AWS +# instance, a developer's own cloud workstation included, so it cannot +# distinguish a buildserver from the machines this is meant to protect. The +# variant can. Both values come from delphix-platform, which ships these two +# commands, and 'internal-buildserver' is the variant its own provisioning uses +# for the buildserver image. +# +# Honors DISABLE_SYSTEM_CHECK, which any host that is not a buildserver image +# needs, a plain Ubuntu AWS instance included. +# +function check_host_is_disposable() { + local platform variant + + if [[ "$DISABLE_SYSTEM_CHECK" == "true" ]]; then + return 0 + fi + + # + # Absence of the commands means this is not a Delphix image at all, which is + # reported separately from being the wrong one: the two call for different + # corrections, and 'command not found' from the assignments below would say + # neither. + # + if ! command -v get-appliance-platform >/dev/null || + ! command -v get-appliance-variant >/dev/null; then + die "This host is not a Delphix appliance image" \ + "(get-appliance-platform and get-appliance-variant are not" \ + "present); refusing to bootstrap a container image on it. Clone" \ + "the dlpx-internal-buildserver-develop group on DCoA, or set" \ + "DISABLE_SYSTEM_CHECK=true to override, at your own risk." + fi + + platform=$(get-appliance-platform) || + die "Could not determine this host's appliance platform." + variant=$(get-appliance-variant) || + die "Could not determine this host's appliance variant." + + if [[ "$platform" != "aws" ]] || [[ "$variant" != "internal-buildserver" ]]; then + die "This is a Delphix '$platform'/'$variant' image, not an" \ + "aws/internal-buildserver one; refusing to bootstrap a container" \ + "image on it. Set DISABLE_SYSTEM_CHECK=true to override, at your" \ + "own risk." + fi +} + +# +# Verify the host can build and run the build container. Note that none of these +# prerequisites reference an Ubuntu codename; that independence is the point. +# +function container_check_host_prereqs() { + command -v docker >/dev/null || + die "docker is not installed; it is required to build packages." + docker info >/dev/null 2>&1 || + die "Cannot talk to the docker daemon. Is it running, and is $USER in the docker group?" + command -v debootstrap >/dev/null || + die "debootstrap is not installed; run 'sudo apt-get install debootstrap'." + sudo -n true 2>/dev/null || + die "Passwordless sudo is required to bootstrap the build container's root filesystem." +} + +# +# Bootstrap a root filesystem for UBUNTU_DISTRIBUTION from the primary package +# mirror and import it as a docker image, then layer on the few things the build +# needs before setup.sh can run inside: sudo, CA certificates, and a user whose +# uid and gid match the invoking user so that artifacts written to the +# bind-mounted checkout are owned correctly on the host. +# +function container_build_base_image() { + check_env TOP UBUNTU_DISTRIBUTION + local tag rootfs tarball mirror script_dir build_root_dir mount_opts + local dockerfile dockerfile_sha + local built_mirror built_uid built_gid built_dockerfile_sha + + logmust container_image_tag + tag="$_RET" + + logmust resolve_mirror_urls + mirror="$_RET_MIRROR_MAIN" + + dockerfile="$TOP/resources/Dockerfile.build-container" + dockerfile_sha=$(sha256sum "$dockerfile" | cut -d' ' -f1) || + die "Could not hash $dockerfile" + + # + # The image is a cache, and its tag alone does not identify what is in it. + # Three things it is keyed on are recorded as labels when it is built, and a + # difference in any of them has to force a rebuild: + # + # - The mirror snapshot its rootfs was bootstrapped from. apt is repointed at + # the resolved snapshot on every build, so an image kept past a snapshot + # advance builds new code against a frozen debootstrap toolchain, which is + # exactly the drift this container was introduced to eliminate, just from + # the other side. + # - The uid and gid it was built for. Those are baked into the image's + # /etc/passwd, and the container runs --user :, so a second user + # on a shared buildserver reusing the first user's image runs as a uid + # with no passwd entry, and sudo fails with "you do not exist in the + # passwd database". + # - The Dockerfile that produced it, by content hash. Docker's own layer cache + # cannot help here, because this check returns before docker build is ever + # reached, so an edit to the Dockerfile would otherwise leave every existing + # image indefinitely stale. What makes that more than untidiness is + # CONTAINER_MARKER_FILE: the Dockerfile is what puts it in the image, and + # running_in_container() is false without it, so an image predating it would + # send the in-container setup.sh down the host path, to bootstrap a + # container from inside one and fail in check_host_is_disposable() with a + # message about the wrong machine entirely. + # + if [[ "$LINUX_PKG_REBUILD_IMAGE" != "true" ]] && + docker image inspect "$tag" >/dev/null 2>&1; then + built_mirror=$(docker image inspect --format \ + '{{index .Config.Labels "com.delphix.linux-pkg.mirror-main"}}' "$tag") + built_uid=$(docker image inspect --format \ + '{{index .Config.Labels "com.delphix.linux-pkg.uid"}}' "$tag") + built_gid=$(docker image inspect --format \ + '{{index .Config.Labels "com.delphix.linux-pkg.gid"}}' "$tag") + built_dockerfile_sha=$(docker image inspect --format \ + '{{index .Config.Labels "com.delphix.linux-pkg.dockerfile-sha256"}}' "$tag") + + if [[ "$built_mirror" == "$mirror" ]] && + [[ "$built_uid" == "$(id -u)" ]] && + [[ "$built_gid" == "$(id -g)" ]] && + [[ "$built_dockerfile_sha" == "$dockerfile_sha" ]]; then + echo "Image $tag already exists and matches the current mirror" \ + "snapshot, user and Dockerfile; skipping bootstrap. Set" \ + "LINUX_PKG_REBUILD_IMAGE=true to force a rebuild." + return 0 + fi + + echo_bold "Rebuilding image $tag: it was built for mirror" \ + "'${built_mirror:-unknown}', uid/gid" \ + "'${built_uid:-unknown}:${built_gid:-unknown}' and Dockerfile" \ + "'${built_dockerfile_sha:-unknown}', but this build needs mirror" \ + "'$mirror', uid/gid '$(id -u):$(id -g)' and Dockerfile" \ + "'$dockerfile_sha'." + fi + + # + # Fail with a specific message when the suite is absent from the mirror, + # since during an LTS upgrade that means the mirror sync has not completed + # yet rather than that anything is broken. Checked before the suite ever + # touches /usr/share below, so a typo can't leave a permanent root-owned + # symlink behind. + # + if ! curl -fsS -o /dev/null "$mirror/dists/$UBUNTU_DISTRIBUTION/Release"; then + die "Suite '$UBUNTU_DISTRIBUTION' is not present in the package mirror" \ + "at $mirror. If this is a new Ubuntu release, the mirror sync for" \ + "it has not completed; see the linux_package_mirror_sync job." + fi + + # + # debootstrap looks up a per-suite script, and ships nothing for a suite + # released after the version installed here, so an older host cannot + # bootstrap a newer suite until it is given one. The symlink is that script. + # + # It points at 'gutsy' because that is the newest Ubuntu suite debootstrap + # carries a real script for: Ubuntu 7.10 was the last release to need its + # own bootstrap logic, and every release since has been compatible with it, + # so debootstrap ships each new suite as a symlink to gutsy rather than a new + # script. Every Ubuntu suite present in a current debootstrap is such a + # symlink (lunar, mantic and noble in 1.0.134ubuntu1), which means this is + # not an approximation of what the suite needs; it is the same script the + # distro itself would have shipped for it. + # + script_dir="/usr/share/debootstrap/scripts" + if [[ ! -e "$script_dir/$UBUNTU_DISTRIBUTION" ]]; then + echo "Teaching debootstrap about $UBUNTU_DISTRIBUTION." + logmust sudo ln -s gutsy "$script_dir/$UBUNTU_DISTRIBUTION" + fi + + # + # The rootfs and its tarball are a few hundred megabytes each -- the imported + # base images measure 258 MB for noble and 284 MB for resolute -- and both + # are untracked, so they are kept out of the checkout, which is bind-mounted + # wholesale into the container. Overridable because debootstrap refuses to + # install into a target mounted nodev or noexec, and on the buildserver this + # was developed against, /tmp and /var/tmp are both mounted that way, which + # rules out the obvious default; whether that holds for every Delphix + # buildserver was not checked, which is why the two checks below test the + # directory actually in use rather than assuming anything about it. + # + build_root_dir="${LINUX_PKG_BUILD_ROOT_DIR:-/var/lib/linux-pkg}" + logmust sudo mkdir -p "$build_root_dir" + + mount_opts=",$(findmnt -no OPTIONS --target "$build_root_dir")," + if [[ "$mount_opts" == *,nodev,* ]]; then + die "$build_root_dir is mounted 'nodev', which debootstrap refuses" \ + "to install into. Set LINUX_PKG_BUILD_ROOT_DIR to a directory" \ + "on a filesystem without that option." + fi + if [[ "$mount_opts" == *,noexec,* ]]; then + die "$build_root_dir is mounted 'noexec', which debootstrap refuses" \ + "to install into. Set LINUX_PKG_BUILD_ROOT_DIR to a directory" \ + "on a filesystem without that option." + fi + + rootfs="$build_root_dir/build-root-$UBUNTU_DISTRIBUTION" + tarball="${rootfs}.tar" + logmust sudo rm -rf "$rootfs" + logmust sudo debootstrap --variant=buildd \ + --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg \ + "$UBUNTU_DISTRIBUTION" "$rootfs" "$mirror" + + # + # Note that tar is not piped into docker import here. logmust dies on + # failure and a piped logmust would die inside a subshell, which would + # lose the error, so the tarball goes to disk and each step is checked. + # + logmust sudo tar -C "$rootfs" -cf "$tarball" . + logmust docker import "$tarball" "${tag}-base" + logmust sudo rm -f "$tarball" + logmust sudo rm -rf "$rootfs" + + # + # The labels are what the cache check above reads back; see the comment + # there for why each one invalidates the image. + # + logmust docker build -t "$tag" \ + --build-arg "BASE=${tag}-base" \ + --build-arg "UID=$(id -u)" \ + --build-arg "GID=$(id -g)" \ + --label "com.delphix.linux-pkg.mirror-main=$mirror" \ + --label "com.delphix.linux-pkg.uid=$(id -u)" \ + --label "com.delphix.linux-pkg.gid=$(id -g)" \ + --label "com.delphix.linux-pkg.dockerfile-sha256=$dockerfile_sha" \ + -f "$dockerfile" "$TOP/resources" +} + +# +# Variables that cross into the build container. This is an allowlist rather +# than a pass-through of the environment, so that the boundary's contract stays +# visible and a missing variable fails loudly instead of silently. +# +# DRYRUN and the PUSH_GIT_* credentials belong here even though no build reads +# them: sync-with-upstream.sh and push-merge.sh re-exec into the container too, +# and both check DRYRUN (and, through push_to_remote(), the credentials) after +# the re-exec, so without these the auto-update jobs that drive them would die +# on every invocation no matter what the host set. +# +# Only variables cross this boundary, never files, so anything naming a path +# needs a bind mount as well; see SECRET_DB_JUMP_BOX_PRIVATE_KEY and ~/.aws +# below. +# +CONTAINER_ENV_ALLOWLIST=" +DEFAULT_GIT_BRANCH +DEFAULT_REVISION +DELPHIX_RELEASE_VERSION +DELPHIX_SIGNATURE_VERSION +DELPHIX_SIGNATURE_URL +DELPHIX_SIGNATURE_TOKEN +DELPHIX_PACKAGE_MIRROR_MAIN +DELPHIX_PACKAGE_MIRROR_SECONDARY +DEPENDENCIES_BASE_URL +JENKINS_OPS_DIR +TARGET_KERNEL_FLAVORS +UBUNTU_DISTRIBUTION +FETCH_GIT_TOKEN +DRYRUN +PUSH_GIT_TOKEN +PUSH_GIT_USER +PUSH_GIT_PASSWORD +" + +# +# Read one package's PACKAGE_NEEDS_DOCKER out of its config.sh, without loading +# the rest of its configuration. Sets _RET. +# +# container_run_args() has to know whether the host's docker socket needs binding +# in before the container is started, which is before load_package_config() runs: +# that runs inside the container, and deliberately, since it resets the hook +# namespace, resolves the package's dependencies and validates its config, none +# of which belongs in the host-side process. So read only this one variable here, +# with the config sourced in a subshell so nothing else it defines or sets can +# leak into the host-side shell, and its output discarded so a config that echoes +# does not interleave with the host's log. A config that cannot be sourced yields +# an empty value rather than an error; the in-container load_package_config() +# reports that properly, with all of its validation in place. +# +function package_needs_docker() { + local pkg="$1" + + check_env TOP + _RET=$( + cd "$TOP" || exit 1 + source "packages/$pkg/config.sh" >/dev/null 2>&1 + echo "$PACKAGE_NEEDS_DOCKER" + ) +} + +# +# Assemble the docker arguments for a build container. Sets _RET_LIST. +# +function container_run_args() { + check_env TOP + local tag var name + + logmust container_image_tag + tag="$_RET" + + _RET_LIST=( + "--init" + "--network" "host" + "--user" "$(id -u):$(id -g)" + "-v" "$TOP:$TOP" + "-w" "$TOP" + ) + + for var in $CONTAINER_ENV_ALLOWLIST; do + [[ -n "${!var+x}" ]] && _RET_LIST+=("-e" "$var") + done + + # + # AWS credentials, secretDb settings, and the per-package overrides are + # named by pattern rather than individually, since the latter are spelled + # with the package's own name (CLOUD_INIT_GIT_BRANCH selects the branch for + # cloud-init, and so on) and so cannot be enumerated here. + # + # Note that these are the per-package *inputs*, read by + # get_package_config_from_env(). The PACKAGE_GIT_URL, PACKAGE_GIT_BRANCH and + # PACKAGE_REVISION that function computes from them deliberately do not + # cross: they are recomputed in the container, where + # reset_package_config_variables() unsets them first anyway. + # + while IFS='=' read -r name _; do + # + # Skip what the allowlist already names, so that a variable matching + # one of the patterns below (DEFAULT_GIT_BRANCH, DEFAULT_REVISION) is + # not passed twice. + # + [[ "$CONTAINER_ENV_ALLOWLIST" == *$'\n'"$name"$'\n'* ]] && continue + + case "$name" in + AWS_* | SECRET_DB_* | *_S3_URL | *_GIT_URL | *_GIT_BRANCH | *_REVISION) + _RET_LIST+=("-e" "$name") + ;; + esac + done < <(env) + + # + # SECRET_DB_JUMP_BOX_PRIVATE_KEY holds a path to an SSH key on the host, so + # the path has to resolve inside the container as well. Docker's classic -v + # syntax does not refuse a missing host source; it silently creates an empty, + # root-owned directory there instead, so without this check a typo here would + # mount an empty directory where the build expects an SSH key, and the failure + # would surface confusingly deep inside the secretDb client rather than here. + # + if [[ -n "$SECRET_DB_JUMP_BOX_PRIVATE_KEY" ]]; then + [[ -f "$SECRET_DB_JUMP_BOX_PRIVATE_KEY" ]] || + die "SECRET_DB_JUMP_BOX_PRIVATE_KEY=$SECRET_DB_JUMP_BOX_PRIVATE_KEY is not" \ + "a regular file; a docker bind mount would silently create an empty" \ + "directory there instead of the expected SSH key." + _RET_LIST+=("-v" + "${SECRET_DB_JUMP_BOX_PRIVATE_KEY}:${SECRET_DB_JUMP_BOX_PRIVATE_KEY}:ro") + fi + + # + # An AWS profile or credentials file only exists as a file, so + # SECRET_DB_AWS_PROFILE crossing the boundary is not enough on its own, and + # neither is anything else that resolves credentials that way: both + # fetch_dependencies() and sign_modules()' key fetch need working AWS + # credentials for most of the packages here. Bind-mounting the directory + # read-only is preferred over asserting on env-var credentials because it + # leaves both credential styles working, and read-only because the container + # has no business rewriting the host's AWS configuration. Mounted at the + # container user's home, not the host user's: only $TOP is mapped through at + # its host path, and $HOME inside the container is the home of the user + # resources/Dockerfile.build-container creates. + # + if [[ -d "$HOME/.aws" ]]; then + _RET_LIST+=("-v" "$HOME/.aws:$CONTAINER_HOME/.aws:ro") + fi + + if [[ "$PACKAGE_NEEDS_DOCKER" == "true" ]]; then + [[ -S /var/run/docker.sock ]] || + die "PACKAGE_NEEDS_DOCKER is set but /var/run/docker.sock does not exist" \ + "on the host; a docker bind mount would silently create an empty" \ + "directory there instead of the socket." + # + # The socket is mode 660 root:docker, and docker does not propagate the + # invoking user's supplementary groups, so a container started with + # --user : gets EACCES on it even when the host user is + # in the docker group. The image's /etc/group has no entry for that gid + # either, so the grant has to come from --group-add, derived from the + # socket itself rather than hardcoded: the docker group's gid differs + # between hosts, and what matters is the group that owns this socket. + # + _RET_LIST+=("-v" "/var/run/docker.sock:/var/run/docker.sock") + _RET_LIST+=("--group-add" "$(stat -c %g /var/run/docker.sock)") + fi + + _RET_LIST+=("$tag") +} + +# +# Start an interactive shell in a fresh build container, with the same mounts +# and environment a build would get. Used to reproduce a build environment +# without reconstructing the docker invocation by hand. +# +function container_shell() { + local -a args tty_opts + + logmust check_host_is_disposable + logmust container_check_host_prereqs + logmust container_build_base_image + logmust container_run_args + args=("${_RET_LIST[@]}") + + # + # -i is unconditional so a piped command list still works; -t is added + # only when stdin is actually a terminal, since docker refuses -t when + # it is not ("the input device is not a TTY"), which would otherwise + # break this flag under a script or CI job feeding it commands. + # + tty_opts=("-i") + [[ -t 0 ]] && tty_opts+=("-t") + + echo_bold "Starting a shell in the build container." \ + "The checkout is bind-mounted at $TOP." + echo_bold "This container has NOT run setup.sh, so it has no build tooling" \ + "yet (no debhelper, no apt sources for $UBUNTU_DISTRIBUTION, no aws)." \ + "Run ./setup.sh inside it first if you need any of that." + exec docker run --rm "${tty_opts[@]}" "${args[@]}" bash +} + +# +# Re-run the calling script inside the build container and exit with its status. +# A no-op when already inside, or when the caller has opted out. +# +function container_reexec() { + local script="$1" + shift + local name rc + local -a args + + running_in_container && return 0 + logmust check_host_is_disposable + if [[ "$LINUX_PKG_NO_CONTAINER" == "true" ]]; then + echo_bold "LINUX_PKG_NO_CONTAINER is set; running on the host." + return 0 + fi + + logmust container_check_host_prereqs + logmust container_build_base_image + logmust container_run_args + args=("${_RET_LIST[@]}") + + name="linux-pkg-$(basename "${script%.sh}")" + [[ -n "$PACKAGE" ]] && name="linux-pkg-$PACKAGE" + + # + # Plain docker rather than logmust: a leftover container from a previous + # failed run is expected, and logmust would die on the removal of one + # that does not exist. + # + docker rm -f "$name" >/dev/null 2>&1 || true + + # + # Not exec'ing here is deliberate: the failure path keeps the container + # around for debugging, which requires a process left to make that + # decision. The trap covers an aborted Jenkins job. + # + trap 'docker rm -f "$name" >/dev/null 2>&1; exit 130' INT TERM + + # + # Credentials embedded in a URL are masked before the command line is + # echoed, the same way push_to_remote() masks them (lib/common.sh): the + # arguments are echoed verbatim otherwise, and '-g https://@github.com/...' + # would then land in a Jenkins console log. Only the URL shape is masked; + # tokens passed by name (-e FETCH_GIT_TOKEN) never appear here to begin with. + # + echo_bold "Running in container:" \ + "$(sed -E 's#(https?://)[^/@[:space:]]+@#\1******@#g' \ + <<<"docker run --name $name ${args[*]} $script $*")" + rc=0 + docker run --name "$name" "${args[@]}" "$script" "$@" || rc=$? + trap - INT TERM + + if [[ $rc -eq 0 ]]; then + docker rm -f "$name" >/dev/null || + die "Could not remove container '$name'" + else + echo_error "Build failed in container '$name', which has been left" \ + "in place for debugging. To inspect it:" + echo_bold " docker start -ai $name" + echo_bold " docker exec -it $name bash # if it is running" + echo_bold "Or to start a fresh container with the same mounts and" \ + "environment, then drive the build by hand from there. That" \ + "container starts without any build tooling, so run ./setup.sh" \ + "inside it first:" + echo_bold " ./buildpkg.sh -S ${PACKAGE:-}" + fi + exit $rc +} diff --git a/packages/containerized-masking/config.sh b/packages/containerized-masking/config.sh index 6c4c166e..36b06aca 100644 --- a/packages/containerized-masking/config.sh +++ b/packages/containerized-masking/config.sh @@ -28,11 +28,37 @@ source "$PWD/lib/common.sh" DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/dms-core-gate.git" +# +# The gradle task below reaches :tools:docker:buildLocalDockerImage, which builds +# a docker image and so needs a docker daemon. The build container gets the +# host's daemon through its socket rather than running one of its own. +# +# Without the socket the failure misdirects: the gradle-docker plugin defaults to +# the unix socket only when one is present and otherwise falls back to TCP, so it +# reports 'Connect to http://127.0.0.1:2375 failed: Connection refused' rather +# than anything about a missing socket. +# +PACKAGE_NEEDS_DOCKER="true" MEND_SCAN_APPLICABLE="true" MEND_SCAN_IMAGES="'delphix-masking-proxy', 'delphix-masking-database', 'delphix-masking-app'" SKIP_COPYRIGHTS_CHECK=true +function prepare() { + # + # Same list, from the same repo, that the 'masking' package installs. + # Without it no JDK is present at all: the JAVA_HOME below names a java-8 + # path that only ever existed because the buildserver happened to have + # one, and dms-core-gate's own gradlew wrapper resets JAVA_HOME to Java 17 + # only when it can find a 17 JDK to switch to. That wrapper is why + # 'masking' builds with the identical stale JAVA_HOME, so installing the + # same dependencies here fixes this package the same way rather than + # introducing a second, different convention. + # + logmust read_list "$WORKDIR/repo/packaging/build-dependencies" + logmust install_pkgs "${_RET_LIST[@]}" +} + function build() { export JAVA_HOME JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/" diff --git a/packages/docker-python-image/config.sh b/packages/docker-python-image/config.sh index a27ed564..b80e6172 100644 --- a/packages/docker-python-image/config.sh +++ b/packages/docker-python-image/config.sh @@ -17,6 +17,12 @@ # shellcheck disable=SC2034 DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/docker-python-image.git" +# +# debian/rules' override_dh_install runs 'docker pull' to fetch the python +# image it repackages, so the build needs a docker daemon. The build container +# gets the host's daemon through its socket rather than running one of its own. +# +PACKAGE_NEEDS_DOCKER="true" function prepare() { logmust install_build_deps_from_control_file diff --git a/packages/gdb-python/config.sh b/packages/gdb-python/config.sh index df5e2beb..014215f0 100644 --- a/packages/gdb-python/config.sh +++ b/packages/gdb-python/config.sh @@ -17,8 +17,16 @@ # shellcheck disable=SC2034 DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/gdb-python.git" +PACKAGE_DEPENDENCIES="libkdumpfile" function prepare() { + # + # debian/control build-depends on libkdumpfile, which is built by + # linux-pkg rather than published in the Ubuntu archive, so apt cannot + # find it and mk-build-deps below fails unless it is installed first. + # Declaring it in PACKAGE_DEPENDENCIES is what puts it in DEPDIR. + # + logmust install_pkgs "$DEPDIR"/libkdumpfile/*.deb logmust install_build_deps_from_control_file } diff --git a/packages/virtualization/config.sh b/packages/virtualization/config.sh index 59c9af8d..e7fe6b3a 100644 --- a/packages/virtualization/config.sh +++ b/packages/virtualization/config.sh @@ -20,6 +20,12 @@ source "$PWD/lib/common.sh" DEFAULT_PACKAGE_GIT_URL="https://github.com/delphix/dlpx-app-gate.git" PACKAGE_DEPENDENCIES="crypt-blowfish host-jdks" +# +# The build packages a docker image (see '-Ddockerize=true' below), which needs +# a docker daemon. The build container gets the host's daemon through its +# socket rather than running one of its own. +# +PACKAGE_NEEDS_DOCKER="true" MEND_SCAN_APPLICABLE="true" function prepare() { diff --git a/packages/zfs/config.sh b/packages/zfs/config.sh index 232fc547..bd1edce5 100644 --- a/packages/zfs/config.sh +++ b/packages/zfs/config.sh @@ -23,6 +23,16 @@ UPSTREAM_GIT_URL="https://github.com/openzfs/zfs.git" UPSTREAM_GIT_BRANCH="master" function prepare() { + # + # configure autodetects systemd support by probing for the systemctl + # binary (config/user-systemd.m4), which is always present on a host that + # runs systemd and absent from a debootstrapped root; that root carries + # libsystemd0 and libudev1, but not systemd itself. When the probe fails + # configure builds '--without systemd', the unit and preset files are + # never generated, and dh_install aborts on the paths debian/*.install + # lists unconditionally; e.g. 'zfs-zed missing files: + # lib/systemd/system/zfs-zed.service'. + # logmust install_pkgs \ alien \ autoconf \ @@ -55,6 +65,7 @@ function prepare() { pkg-config \ po-debconf \ python3 \ + systemd \ uuid-dev \ zlib1g-dev logmust install_kernel_headers_and_dbgsyms diff --git a/push-merge.sh b/push-merge.sh index 8afe280a..1b7de8f9 100755 --- a/push-merge.sh +++ b/push-merge.sh @@ -17,10 +17,9 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" +ORIGINAL_ARGS=("$@") check_env DEFAULT_GIT_BRANCH -logmust check_running_system -logmust run_setup_if_needed function usage() { [[ $# != 0 ]] && echo "$(basename "$0"): $*" @@ -50,6 +49,10 @@ shift $((OPTIND - 1)) [[ $# -gt 1 ]] && usage "too many arguments" >&2 PACKAGE=$1 +logmust container_reexec "./$(basename "$0")" "${ORIGINAL_ARGS[@]}" +logmust check_running_system +logmust run_setup_if_needed + if [[ "$DRYRUN" != 'false' ]]; then die "DRYRUN environment variable must be set to 'false'." fi diff --git a/resources/Dockerfile.build-container b/resources/Dockerfile.build-container new file mode 100644 index 00000000..ebea7f4d --- /dev/null +++ b/resources/Dockerfile.build-container @@ -0,0 +1,62 @@ +# Copyright 2026 Delphix +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ARG BASE +FROM ${BASE} + +ARG UID +ARG GID + +# lsb-release is not guaranteed to be present under --variant=buildd, and +# check_running_system() depends on lsb_release being available. +# +# gnupg has no known consumer here: configure_apt_sources() names the secondary +# mirror's key with signed-by= and apt verifies that signature with gpgv, which +# apt depends on directly. It is kept anyway, because confirming it is +# unnecessary needs a full build from an image built without it, and the window +# it covers -- before the first successful apt-get update, when nothing can be +# installed on demand -- is where being wrong is most expensive. Drop it once +# such a build has passed. +# +# locales, plus the locale-gen and ENV below, give the container the UTF-8 locale +# the buildserver supplied ambiently. This is environment rather than per-package +# tooling, and it is set here rather than in a package's config.sh because the +# host gave it to all 38 packages equally: windows-connector never exports LANG, +# yet its javac compiles UTF-8 source comments on the host and emits +# "unmappable character ... for encoding US-ASCII" in a container without this. +# Getting a locale wrong is also silent rather than loud, changing encoding, sort +# order and date formatting instead of failing, so it is the wrong thing to make +# each package remember to ask for. +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + sudo ca-certificates curl lsb-release gnupg locales && \ + locale-gen en_US.UTF-8 && \ + rm -rf /var/lib/apt/lists/* + +# Declared alongside the generation above so the two cannot drift apart: a +# generated locale nothing selects leaves glibc on C/POSIX, which is the +# half-fix this replaces. en_US.UTF-8 specifically, because that is what +# packages/virtualization's build() already exports LANG to. +ENV LANG=en_US.UTF-8 + +# lib/container.sh's running_in_container() keys off this file's presence to tell +# an in-container process from a host one. It lives in the image, rather than +# being an environment variable the run command passes in, so that a host cannot +# claim to be this container; see CONTAINER_MARKER_FILE there. +RUN echo "linux-pkg build container" >/etc/linux-pkg-build-container + +RUN groupadd -o -g "${GID}" delphix && \ + useradd -o -m -u "${UID}" -g "${GID}" -s /bin/bash delphix && \ + echo 'delphix ALL=(ALL) NOPASSWD:ALL' >/etc/sudoers.d/delphix && \ + chmod 0440 /etc/sudoers.d/delphix diff --git a/setup.sh b/setup.sh index 4b106813..7e9ab749 100755 --- a/setup.sh +++ b/setup.sh @@ -18,39 +18,56 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" -logmust check_running_system check_env DEFAULT_GIT_BRANCH DELPHIX_RELEASE_VERSION +# +# setup.sh has two jobs, selected by where it runs. On the host it prepares the +# host and builds the container image that package builds run inside. Inside that +# container it configures apt and installs the build tooling, which is what the +# rest of this file does. LINUX_PKG_NO_CONTAINER=true opts a caller out of the +# container entirely (see container_reexec()), so in that case this host-only +# branch must be skipped as well, or the escape hatch would bootstrap a container +# image it never uses and exit before installing the build tooling it does need. +# +# check_host_is_disposable must run for every non-container invocation, not just +# the one that goes on to bootstrap a container image: LINUX_PKG_NO_CONTAINER=true +# skips the block below entirely and falls through to the in-container body, +# which on a bare invocation of this script is not actually inside a container at +# all, and would rewrite this host's /etc/apt/sources.list and install packages +# onto it. Skipped only when already running_in_container, where the question of +# whether the host is disposable no longer applies. +# +if ! running_in_container; then + logmust check_host_is_disposable +fi + +# +# Swap is not arranged here. Package builds spike memory (a linux-kernel or a +# zfs object-agent link will), and the swap that absorbs it belongs to the build +# host: delphix-platform provisions 8G on /dev/nvme1n1 for the +# internal-buildserver variant, re-enabled on each clone's first boot. The host +# is required to be that variant, so it is always present. +# +if ! running_in_container && [[ "$LINUX_PKG_NO_CONTAINER" != "true" ]]; then + logmust container_check_host_prereqs + logmust container_build_base_image + exit 0 +fi + # # Update the sources.list file to point to our internal package mirror. If no # mirror url is passed in, then the latest mirror snapshot is used. # function configure_apt_sources() { - local package_mirror_url - local primary_url="$DELPHIX_PACKAGE_MIRROR_MAIN" - local secondary_url="$DELPHIX_PACKAGE_MIRROR_SECONDARY" - - if [[ -z "$primary_url" ]] || [[ -z "$secondary_url" ]]; then - local latest_url - local delphix_version="$DELPHIX_RELEASE_VERSION" - if compare_versions "$delphix_version" eq "9999.0.0.0" || compare_versions "$delphix_version" gt "2025.3"; then - latest_url="http://linux-package-mirror-v2.delphix.com/" - else - latest_url="http://linux-package-mirror.delphix.com/" - fi - - if is_release_branch; then - package_mirror_url="${latest_url}releases/${DELPHIX_RELEASE_VERSION}" - else - latest_url+="${DEFAULT_GIT_BRANCH}/latest/" - package_mirror_url=$(curl -LfSs -o /dev/null -w '%{url_effective}' \ - "$latest_url" || die "Could not curl $latest_url") - # Remove trailing slash, if present. - package_mirror_url="${package_mirror_url%/}" - fi - [[ -z "$primary_url" ]] && primary_url="${package_mirror_url}/ubuntu" - [[ -z "$secondary_url" ]] && secondary_url="${package_mirror_url}/ppas" - fi + local primary_url + local secondary_url + local secondary_keyring + + secondary_keyring=/etc/apt/keyrings/delphix-secondary-mirror.gpg + + logmust resolve_mirror_urls + primary_url="$_RET_MIRROR_MAIN" + secondary_url="$_RET_MIRROR_SECONDARY" # # Store the package mirror in a file so that it can be added to a @@ -67,6 +84,31 @@ function configure_apt_sources() { die "Could not remove /etc/apt/sources.list.d" ) + # + # The secondary mirror's key, installed before the sources that reference it + # so that the path on its source line resolves by the time apt reads it. + # + # The key is named with signed-by= on the one source line it belongs to, so it + # can only ever validate that repository. Installing it globally instead would + # give it authority over every source in this file, the primary Ubuntu mirror + # lines below included, which is more than the secondary mirror needs; apt-key + # is also absent from Ubuntu 26.04 ("resolute") and later. The key at rest is a + # binary, dearmored OpenPGP keyring, which is what signed-by wants; + # /etc/apt/keyrings may not exist in a freshly debootstrapped rootfs, hence + # install -D. + # + logmust sudo install -D -o root -g root -m 0644 \ + "$TOP/resources/delphix-secondary-mirror.key" \ + "$secondary_keyring" + + # + # Scoping the key above is pointless while a globally trusted copy of it sits + # in trusted.gpg.d, so that path is cleared. It only ever holds one on the + # LINUX_PKG_NO_CONTAINER escape hatch, where the host's own apt configuration + # persists between runs; a container is bootstrapped fresh and never has it. + # + logmust sudo rm -f /etc/apt/trusted.gpg.d/delphix-secondary-mirror.gpg + sudo bash -c "cat <<-EOF >/etc/apt/sources.list deb ${primary_url} ${UBUNTU_DISTRIBUTION} main restricted universe multiverse deb-src ${primary_url} ${UBUNTU_DISTRIBUTION} main restricted universe multiverse @@ -80,42 +122,21 @@ function configure_apt_sources() { deb ${primary_url} ${UBUNTU_DISTRIBUTION}-backports main restricted universe multiverse deb-src ${primary_url} ${UBUNTU_DISTRIBUTION}-backports main restricted universe multiverse - deb ${secondary_url} ${UBUNTU_DISTRIBUTION} main multiverse universe stable + deb [signed-by=${secondary_keyring}] ${secondary_url} ${UBUNTU_DISTRIBUTION} main multiverse universe stable EOF" || die "/etc/apt/sources.list could not be updated" - - logmust sudo apt-key add "$TOP/resources/delphix-secondary-mirror.key" } # -# Some packages require cause a spike in memory usage during the build, so -# we add a swap file to prevent the oom-killer from terminating the build. +# Everything below rewrites this root filesystem's apt sources to the suite named +# by UBUNTU_DISTRIBUTION and installs from them, so the codename of the system it +# runs on has to match that suite. Inside the container it does by construction, +# and the check catches a stale or mismatched image. On the +# LINUX_PKG_NO_CONTAINER escape hatch there is no container and no such +# guarantee, and this is the only thing standing between, say, a jammy host and +# an apt configuration pointed at noble; the base setup.sh checked the same +# thing, in the same place, for the same reason. # -function add_swap() { - local rootfs - local swapfile - - swapfile="/swapfile" - rootfs=$(awk '$2 == "/" { print $3 }' /proc/self/mounts) - - # - # If the root filesystem is ZFS, we assume we're running on a - # Delphix based buildserver, and assume swap is already enabled; - # the Delphix buildserver should enable swap for us. - # - if [[ "$rootfs" == "zfs" ]]; then - return - fi - - # Swap already enabled, nothing to do. - if sudo swapon --show | grep -q "$swapfile"; then - return - fi - - logmust sudo fallocate -l 4G "$swapfile" - logmust sudo chmod 600 "$swapfile" - logmust sudo mkswap "$swapfile" - logmust sudo swapon "$swapfile" -} +logmust check_running_system logmust configure_apt_sources logmust sudo apt-get update @@ -128,22 +149,98 @@ logmust sudo apt-get update # command. # - equivs is used by the mk-build-deps utility which is used to install # build dependencies from a control file. -# - install_shfmt and shellcheck are needed for - make check - to be able to -# make sure style checks are fine. +# - fakeroot is required by dpkg-buildpackage (dpkg_buildpackage_default(), the +# default build() implementation used by most packages) to fake root +# ownership of package contents without actually running as root. +# - shellcheck, and install_shfmt below, provide the two tools - make check - +# runs. Nothing inside the container runs it as part of a build, and the +# repo's own style gate does not depend on this script either: the GitHub +# workflow installs both itself (.github/scripts/install-shellcheck.sh and +# install-shfmt.sh). They are here so that the style checks can be run against +# the bind-mounted checkout from a container shell (./buildpkg.sh -S), on a +# host whose own release may not even be the one being built. # - jq is used to generate a JSON formatted metadata file by some packages. +# - git is used below to clone every package's repo in its fetch stage, and to +# set the global user.email/user.name just after this install_pkgs call. +# - wget is used by install_shfmt() to fetch the shfmt binary. +# - python3-pip is used below to install awscli, which is not itself part of +# this list: it has no installation candidate on noble ("E: Package +# 'awscli' has no installation candidate" against our mirror), since Ubuntu +# distributes it via pip or snap from 24.04 onward instead of a deb. +# - bc is used by connstat's module/configure.sh to compute the +# KERNEL_CENTEVERSION preprocessor value baked into its Makefile +# (kcentevers=$(echo 100*$major+$minor | bc)); without it that command +# substitution silently yields an empty value, which turns into an empty +# -DKERNEL_CENTEVERSION= compiler flag and a cascade of unrelated-looking +# syntax errors deep in connstat.c's version-gated #if blocks. +# - sbsigntool provides kmodsign, used by sign_modules() to sign kernel +# modules for packages like connstat and zfs. Without it, every module's +# kmodsign call fails with "command not found" inside sign_modules()'s +# find | while read loop; that loop runs in a subshell, so the failure +# is confined to it rather than aborting the build, and the resulting +# package silently ships an unsigned .ko instead of failing loudly. +# - kmod provides modinfo, which sign_modules() also runs (after kmodsign, +# in the same subshelled loop) purely to log the signer's identity. Same +# silent-failure shape as kmodsign above, but this call doesn't mutate +# the module, so its absence doesn't affect the signature itself. +# None of build-essential/debhelper/devscripts/equivs/fakeroot/git/wget are +# part of debootstrap's --variant=buildd set, so they cannot be assumed +# present. # logmust install_pkgs \ build-essential \ debhelper \ devscripts \ equivs \ + fakeroot \ + git \ rsync \ shellcheck \ - jq + wget \ + jq \ + python3-pip \ + bc \ + sbsigntool \ + kmod -logmust install_shfmt +# +# aws is used by fetch_dependencies() to pull a package's build-dependencies +# from S3. It is present on a real buildserver's host root (part of that +# host's own provisioning, outside this repo's control), but the container is +# a separate, debootstrapped rootfs that never gets it any other way. Installed +# here, in the in-container provisioning that runs on every build, rather than +# baked into resources/Dockerfile.build-container, so existing images keep +# working without a rebuild. Matches the precedent in appliance-build's own +# bootstrap (bootstrap/roles/appliance-build.bootstrap/tasks/main.yml), which +# installs it the same way for the same reason. +# +# Pinned to the version validated against this setup, unlike every apt package +# above: those are implicitly pinned through the branch's mirror snapshot, but +# PyPI has no equivalent, so an unpinned install would let a future awscli +# release change the toolchain with no corresponding repo change, and an older +# build could no longer be reproduced. Bump deliberately, not silently. +# +# Retried the same way install_pkgs() retries apt, because this is the one build +# input that does not come from the pinned mirror snapshot: it reaches PyPI over +# the internet on every build of every package, so a transient failure there +# would otherwise fail a build that has nothing wrong with it. +# +function install_awscli() { + local attempt + + for attempt in {1..3}; do + echo "Running: sudo pip3 install awscli==1.45.63 --break-system-packages" + sudo pip3 install "awscli==1.45.63" --break-system-packages && return -logmust add_swap + echo "pip3 install awscli failed, retrying." + sleep 10 + done + die "pip3 install awscli failed after $attempt attempts" +} + +logmust install_awscli + +logmust install_shfmt logmust git config --global user.email "eng@delphix.com" logmust git config --global user.name "Delphix Engineering" diff --git a/sync-with-upstream.sh b/sync-with-upstream.sh index 568052af..17b4cac6 100755 --- a/sync-with-upstream.sh +++ b/sync-with-upstream.sh @@ -17,10 +17,9 @@ TOP="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" source "$TOP/lib/common.sh" +ORIGINAL_ARGS=("$@") check_env DEFAULT_GIT_BRANCH -logmust check_running_system -logmust run_setup_if_needed function usage() { [[ $# != 0 ]] && echo "$(basename "$0"): $*" @@ -52,6 +51,10 @@ shift $((OPTIND - 1)) [[ $# -gt 1 ]] && usage "too many arguments" >&2 PACKAGE=$1 +logmust container_reexec "./$(basename "$0")" "${ORIGINAL_ARGS[@]}" +logmust check_running_system +logmust run_setup_if_needed + logmust check_package_exists "$PACKAGE" merging_ref="refs/heads/projects/auto-update/$DEFAULT_GIT_BRANCH/merging"