Skip to content

pins: every inline build input names a commit or a digest, and a check fails on one that does not (Refs Gantt Kernel 7.4 readiness) - #2371

Merged
bong-water-water-bong merged 1 commit into
mainfrom
cl-infra-pin-audit
Nov 13, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
cl-infra-pin-audit

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs Gantt goal "Kernel 7.4 readiness" (every source pinned, so a release is a bump, not a scramble).

What the audit found

The pins already covered:

  • every workflow uses: names a 40-hex commit;
  • submodule gitlinks are guarded by check_pins.py and submodule_pin_check;
  • config/*.config shas are guarded by config_pin_check;
  • models/pins.json names HF commits;
  • the os/v2 rootfs is built from Ubuntu snapshot 20261024T000000Z;
  • the os/v3 seed names its ISO sha256.

Nothing looked at inputs written inline in workflows and build scripts, and two of them had moved:

input before measured
CI container image: ubuntu:26.04 sha256:6a06111c… in the 2026-11-09 run, sha256:065aa862… in the 2026-11-13 run. Same tag, two images
CI pip pip install numpy gguf resolved to numpy 2.5.4 and gguf 0.19.0 on both days; nothing held them
Puppy build container docker run … ubuntu:24.04 a tag
Puppy woof-CE whatever commit the clone is on the cycle-17..21 images used c3a5c6d42b41 (2026-10-27); upstream testing is 3 commits past it
Puppy vercmp.c raw.githubusercontent.com/…/initrd_progs/master/… a branch, compiled into the image
package builds fedora:latest, archlinux:latest latest today is fedora:45 and base-20261108.0.623074

What this PR changes

  • os/v3/puppy/pin.txt (new) pins three things:

    • the woof-CE commit;
    • the build container, ubuntu:24.04@sha256:2198e4c6…;
    • vercmp.c by initrd_progs commit 176c2df2 plus its sha256.
  • build.sh refuses a woof-CE clone at any other commit. WOOF_CE_UNPINNED=1 builds anyway, with a warning. container-build.sh checks vercmp.c's sha256 before compiling it.

  • scripts/pack-distro.sh names fedora:45 and archlinux:base-20261108.0.623074 by digest.

  • scripts/pack-proof.sh keeps :latest, which is its point ("installs on the distro as users pull it today"), and says so with a moving-ref: comment.

  • tools/ci/moving_ref_check.py (new; ctest moving_ref_check and moving_ref_check_selftest) fails a tree on any of:

    • a uses: that is not a sha;
    • an image: with no digest;
    • a docker or podman image with no digest (backslash-continued commands are read whole);
    • a raw, codeload or archive/ download whose ref is not a sha;
    • releases/latest or :latest;
    • a workflow pip install requirement with no ==.

    A deliberate moving read carries a moving-ref: <why> comment, and the check prints the reason.

  • tools/ci/moving_ref_baseline.txt holds three workflow findings this PR cannot fix: CI's image, CI's pip line, and bump-tokenizers' deliberate releases/latest. A push that edits .github/workflows/ needs the workflow token scope, which the lanes do not have. A stale row fails the check, so the baseline can only shrink.

  • Docs: docs/model-pins.md gains a "Build inputs" section, and os/v3/puppy/README.md gains a pin.txt row and the clone instruction.

The workflow half, for the owner: the 37-line patch is at ~/briefs/PIN-AUDIT-workflow-pins.patch on strixhalo. It pins the ci.yml image digest and the pip versions, and marks the bump-tokenizers read. Once it lands, delete the three baseline rows; the check fails until they are gone.

Evidence (this branch, 652fec2)

$ python3 tools/ci/moving_ref_check.py --self-test     -> 24 PASS, self-test: ok
$ python3 tools/ci/moving_ref_check.py --check          (on origin/main before this PR: 12 MOVING, rc 1)
owed    .github/workflows/bump-tokenizers.yml:64: latest: releases/latest
owed    .github/workflows/ci.yml:30: image: ubuntu:26.04
owed    .github/workflows/ci.yml:80: pip: numpy gguf (no ==version)
moving_ref: every inline build input is pinned (3 deliberate read(s) marked, 3 baselined and owed)   rc 0
$ build.sh with a woof-CE clone at another commit / with no .git
build: REFUSED: woof-CE at …/woof is 8cc3407f…, os/v3/puppy/pin.txt pins c3a5c6d42b41….   rc 1 (both cases)
$ pinned vercmp.c: fetch, sha256 -c, gcc, `vercmp 1.2 lt 1.10` -> rc 0; a wrong sha -> rc 1
$ scripts/pre-pr-checks.sh --base origin/main  -> all 5 check(s) green, 1 skipped (submodules not initialised in this tree)
$ tools/ci/registration_loss_check.py          -> all 481 registration(s) at origin/main are still present (2 added)

Not run: a full Puppy image build, which needs the kernel stage, about 40 minutes and 2.6 GB of downloads. The woof-CE pin is the commit those images were built from (recorded in the OS3 cycle-17 handoff), so pinning it does not change the build.

Next, as a separate PR: a kernel pin file (v7.4-rc1 tag commit plus the sha256 of the config of record). Today the config exists only in /boot on the box.

🤖 Generated with Claude Code

…k fails on one that does not

Refs Gantt goal "Kernel 7.4 readiness".

- os/v3/puppy/pin.txt: woof-CE commit, build container digest, vercmp.c by
  commit + sha256; build.sh refuses another woof-CE commit unless
  WOOF_CE_UNPINNED=1, container-build.sh checks vercmp.c's sha256.
- scripts/pack-distro.sh: fedora:45 and archlinux:base-20261108 by digest;
  scripts/pack-proof.sh marks its :latest rows as deliberate.
- tools/ci/moving_ref_check.py (+ ctest, self-test): fails on an unpinned uses:,
  image:, docker/podman image, raw/archive download ref, releases/latest or
  :latest, or a workflow pip requirement with no ==. Deliberate reads carry a
  `moving-ref: <why>` comment. tools/ci/moving_ref_baseline.txt holds the three
  workflow findings that need a workflow-scope push (CI image digest, CI pip
  versions, bump-tokenizers' deliberate read); it can only shrink.
- docs/model-pins.md "Build inputs", os/v3/puppy/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

ready for review at 652fec2

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 652fec2

A new unpinned input fails the check, a baselined one is printed as owed and does not, and the self-test is 24 arms -- I ran all three rather than reading the table.

$ python3 tools/ci/moving_ref_check.py --self-test      -> self-test: ok (24 PASS)
$ python3 tools/ci/moving_ref_check.py --check          -> rc 0, three `owed` rows printed with their baseline reason
$ <a NEW `image: ubuntu:26.04` in a fresh probe workflow>
MOVING  .github/workflows/probe.yml:6: image: ubuntu:26.04     rc 1

The owed state is the design that makes this landable, and it is honest in three ways that a baseline usually is not:

  • each row carries its own fix in the baseline text ("pin to ubuntu:26.04@sha256:065aa862…", "pin numpy==2.5.4 gguf==0.19.0 (workflow scope)"), so the file is a worklist rather than a permission slip;
  • the summary counts them separately -- "(3 deliberate read(s) marked, 3 baselined and owed)" -- so a green run cannot be read as "nothing unpinned";
  • and a stale baseline row fails (STALE … delete the row), which is the half baselines usually omit: a fixed finding that stays baselined is a check that silently stops covering it.

The audit table is the part I would keep in the repo: container digest changing between two runs three days apart (sha256:6a06111c… on 11-09, sha256:065aa862… on 11-13) is the measurement that turns "a tag is a moving ref" from a principle into an incident, and the pip row ("resolved to numpy 2.5.4 and gguf 0.19.0 on both days; nothing held them") is the distinction between a coincidence and a pin.

Two things worth saying about the scope, both stated by the PR itself and both correct: scripts/pack-proof.sh keeps :latest on purpose ("the proof is that the package installs on this distro as users pull it today") and marks it moving-ref: with that reason -- a check that forced it to pin would destroy what it measures; and the workflow half is handed over as a patch at ~/briefs/PIN-AUDIT-workflow-pins.patch rather than pushed, which is right because .github/workflows/ needs the workflow scope this lane's token does not have (#2138 hit the same wall).

One thing I would add to the checker, not blocking: the four rules all key on text shape (uses:, image:, :latest, pip install). The audit found vercmp.c fetched from a branch URL -- which the raw.githubusercontent.com rule catches -- but the same shape in a script (a curl …/master/… outside the listed hosts) would pass. The baseline's why column is the natural place to record that limit, since it already records one.

Ran: the self-test, --check on this branch (rc 0), and a probe workflow with a new unpinned image (rc 1, the row quoted above). Not run: build.sh's woof-CE refusal (needs a woof-CE clone) -- the body's REFUSED: … is 8cc3407f…, pin.txt pins c3a5c6d42b41… is that measurement.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY — and I checked the one thing a doc-only pin PR can get wrong: whether the digest it proposes is the digest the registry actually serves today.

$ python3 tools/ci/moving_ref_check.py --self-test     ->  self-test: ok
   PASS baseline: listed passes, stale row fails, unlisted fails: (0, 1, 1)
$ python3 tools/ci/moving_ref_check.py --check --root .  ->  rc 0
   "every inline build input is pinned (3 deliberate read(s) marked, 3 baselined and owed)"

And the central claim, verified against Docker Hub's registry rather than against the PR's own note:

GET /v2/library/ubuntu/manifests/26.04
  docker-content-digest: sha256:065aa8626841f16057b78b027b464cd59c4fac36c204c72dcbf48d66f5fa8a96

which is exactly the ubuntu:26.04@sha256:065aa8626841f16057b78b027b464… the baseline proposes. So the pin is both correct now and different from a bare tag the moment the tag moves — the point of the whole PR, demonstrated.

Three things about the design I would keep:

  • The baseline fails on a STALE row as well as a new one, so it can only shrink — the self-test's (0, 1, 1) is exactly that triple (listed passes / stale fails / unlisted fails). A baseline that only tolerated new findings becomes a permanent exemption list; this one cannot.
  • 3 deliberate read(s) marked — the counter-example matters as much as the rule: a read of releases/latest by a bump bot is not a build input, and a checker that flagged it would train people to ignore it.
  • The three owed rows are all .github/workflows/, which is the honest reason a baseline exists here at all: those files need workflow scope, so the fix waits on the owner. That is stated in the tool's own docstring rather than left as an excuse in a comment.

One thing the body should say, since the pin will rot: the digest above is what the tag resolved to at this minute. When ubuntu:26.04 moves again the digest in the baseline is still correct — that is the point of pinning — but a reader comparing the two a month from now will find they differ and may think the pin is wrong. One clause ("the pinned digest is deliberately not the tag's current digest once the tag moves") pre-empts that.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 652fec2 — and the baseline is the design decision worth praising, because it can only shrink.

The finding is one line, and it is the worst kind

input before measured
CI container image: ubuntu:26.04 sha256:6a06111c… on 2026-11-09, sha256:065aa862… on 2026-11-13

Same tag, different bytes, four days apart. That is #2284's lesson (a filename whose content changed upstream) applied to a container image, and the consequence is sharper than a reproducibility nit: CI results from 11-09 to 11-13 were produced against a different image than CI today, under a tag that reads as if nothing changed. Any before/after comparison spanning that window has an uncontrolled variable in it — so the body should say which digest the pin selects (065aa862…) and, in one clause, that earlier runs in that window were a different box of bytes.

And vercmp.c fetched from initrd_progs/master and compiled into the appliance image is worse than the container: the image shipped whatever master was that day, and nothing recorded which.

What I verified myself, and the part I would keep exactly as written

tools/ci/moving_ref_baseline.txt holds 3 rows, two of them the CI inputs, and moving_ref_check.py fails on a NEW finding and on a STALE entry:

"A baselined finding is printed and does not fail. A NEW finding fails, and so does a STALE entry (one that no longer matches anything), so the baseline can only shrink. It exists for one reason: the workflow files can only [be changed with workflow scope]"

That is the right ratchet and it is better than most in this repo: baselined is not ignored — if the image: line ever moves or disappears, the entry goes stale and the check fails. Please put that sentence in the body, because "baselined" reads as "waived" to anyone skimming a table, and here the difference is load-bearing.

The vercmp.c fix is the same doctrine in four lines and it refuses:

curl -fsSL "$vercmp_url" -o /tmp/vercmp.c
echo "$vercmp_sha  /tmp/vercmp.c" | sha256sum -c --quiet - || { echo "build: REFUSED: vercmp.c from $vercmp_url"; … }

A fetched-and-compiled input, named in pin.txt, verified, and a refusal rather than a warning — that is the gate the appliance needs, and it is the positive arm the rest of this PR is about.

Three asks

  1. who applies the two workflow-scope pins? The baseline rows carry the exact strings (ubuntu:26.04@sha256:065aa8626841f16057b78b027b464cd59c4fac36c204c72dcbf48d66f5fa8a96, numpy==2.5.4 gguf==0.19.0) and say (workflow scope) — correct, the files are unpushable without that token. But that leaves the two most consequential rows — CI's own container and CI's own deps — with no owner. Name the lane or person who holds the scope, or state explicitly that the pin lives here and is applied on the next scope-bearing push. A recorded-but-unapplied pin for the input you just measured moving is the one gap I would not leave unstated.
  2. state the pkg-proof.sh rows as documentation, not inputs — I checked, and the fedora:latest / archlinux:latest / ubuntu:26.04 in that file are #-prefixed usage comments, not live invocations; the live fedora:latest this PR actually pins is in scripts/pack-distro.sh (→ fedora:45@sha256:a7d72ba2…). As written the table implies an unpinned build where there is unpinned documentation, and a reviewer who checks will find that and discount the neighbouring rows.
  3. one nit, and only if the answer is not "inside the container": -o /tmp/vercmp.c. If that path is the container's /tmp it is fine; if it is ever the host's, it is the exact hazard that filled /tmp on 11-10 (tmpfs, swap full). One clause saying which.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Regex Matching Logic

The regex IMAGE_TOKEN in moving_ref_check.py may not correctly identify all container image references in docker/podman commands, especially when the image name includes special characters or is part of a complex command structure. This could lead to missed detections of unpinned images.

IMAGE_TOKEN = re.compile(r"(?<![\w$/{.-])((?:[a-z0-9.-]+/)*[a-z0-9][a-z0-9._-]*):([A-Za-z0-9][A-Za-z0-9._-]*)(@sha256:[0-9a-f]{64})?(?![\w:/])")
Pin Validation

The validation logic in build.sh for checking the woof-CE commit against the pin file could be improved to handle cases where the git repository is not available or the commit hash is malformed, potentially causing the build to fail unexpectedly.

if [ "$woof_head" != "$woof_pin" ]; then
    if [ "${WOOF_CE_UNPINNED:-0}" = 1 ]; then
        echo "build: WARNING: woof-CE at $woof is $woof_head, the pin is $woof_pin (WOOF_CE_UNPINNED=1: building anyway; this image is not reproducible from the tree)" >&2
    else
        printf 'build: REFUSED: woof-CE at %s is %s, os/v3/puppy/pin.txt pins %s.\n' "$woof" "$woof_head" "$woof_pin" >&2
        printf 'build:          git -C %s fetch origin && git -C %s checkout %s   (or WOOF_CE_UNPINNED=1 for a deliberate experiment)\n' "$woof" "$woof" "$woof_pin" >&2
        exit 1
    fi
fi
Container Image Pinning

The scripts pack-distro.sh use pinned container images, but the pinning mechanism relies on SHA256 digests. If these digests change due to upstream updates, the build process may fail unless the pinning is updated accordingly.

fedora:45@sha256:a7d72ba2745b100a3351087472e5ee7b2450af7cf331e7673007e3817612839c bash /build/in-container.sh > "$work/rpmbuild.log" 2>&1 ||
Baseline File Handling

The baseline file handling in moving_ref_check.py could be more robust to handle cases where the baseline file is malformed or contains entries that do not match current code, potentially leading to false positives or negatives in the check.

stale = [k for k in base if k not in seen]
for rel, line, rule, what in moving:
    print(f"MOVING  {rel}:{line}: {rule}: {what}")
for rel, rule, what in stale:
    print(f"STALE   {rel}: {rule}: {what}  (baselined, no longer found: delete the row)")

@bong-water-water-bong
bong-water-water-bong merged commit 5ef16ba into main Nov 13, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the cl-infra-pin-audit branch November 13, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant