Skip to content

Pin CI's qpdf and poppler oracles instead of floating on ubuntu-latest - #381

Merged
Tim81 merged 5 commits into
mainfrom
ci/pin-oracle-toolchain
Aug 31, 2026
Merged

Pin CI's qpdf and poppler oracles instead of floating on ubuntu-latest#381
Tim81 merged 5 commits into
mainfrom
ci/pin-oracle-toolchain

Conversation

@Tim81

@Tim81 Tim81 commented Aug 31, 2026

Copy link
Copy Markdown
Owner

What

CI installed qpdf, poppler-utils, fonts-dejavu-core and fonts-texgyre unpinned, so the oracle
suite validated against whatever the runner image happened to ship that day, with no commit to
blame when that changed underneath the repository.

  • Runner: runs-on moves from the floating ubuntu-latest to ubuntu-24.04. Every apt pin
    below is noble-only, and would all break at once the day GitHub retargets the ubuntu-latest
    label to a newer release. aot-smoke stays on ubuntu-latest, noted as deliberate — it carries
    no apt pins, so a label move has nothing there to break.
  • qpdf: apt's version on the runner is 11.9.0, two majors behind upstream 12.4.1.
    --check's output has changed across qpdf majors, and Reader: write out a decrypted copy of an encrypted document #186's acceptance criterion is exactly
    that output. CI now downloads qpdf-12.4.1-bin-linux-x86_64.zip from the official GitHub
    release, verifies its sha256, extracts it to /opt/qpdf-12.4.1, and adds bin/ to PATH plus
    sets QPDF_HOME (the env var tests/VellumPdf.TestSupport/ExternalTool.cs already honors).
  • poppler-utils, fonts-dejavu-core, fonts-texgyre: pinned to the exact versions
    ubuntu-24.04's noble archive serves — measured via apt-cache policy in an ubuntu:24.04
    container: poppler-utils=24.02.0-1ubuntu9.9, fonts-dejavu-core=2.37-8,
    fonts-texgyre=20180621-6. Fonts feed fixture generation, so an unpinned update silently
    changes rendered output for every downstream oracle. The =version pins alone are a scheduled
    outage, though: noble's -updates/-security pockets keep only the newest revision of each
    package (poppler-utils has already rolled -1ubuntu9.1 through -1ubuntu9.8 off the archive),
    so the exact revisions above would eventually 404 with no code change involved. The install
    step now also passes -o APT::Snapshot=$APT_SNAPSHOT on apt-get update/install, which
    adds the dated Ubuntu snapshot service alongside whatever
    sources the runner already has (additive, not a replacement — ubuntu-24.04 resolves its own
    sources through mirror+file:/etc/apt/apt-mirrors.txt rather than a literal
    archive.ubuntu.com host, so the mechanism comment does not assume which host actually serves
    a given fetch). Verified against poppler-utils 24.02.0-1ubuntu9.1, a revision genuinely
    absent from today's live archive: a snapshot dated to when 9.1 was current resolves and
    installs it correctly with the live sources still present. apt-get update reports success
    (exit 0) even when the snapshot fetch itself fails outright — reproduced two ways in a
    container (a TLS trust failure from missing ca-certificates, and a blackholed host) — so the
    step now also greps /var/lib/apt/lists/ for the snapshot's own index files right after and
    fails the build if they never landed, rather than silently continuing on whatever else was
    already cached or configured. CONTRIBUTING.md documents the bump procedure for a genuine
    version bump, including picking an APT_SNAPSHOT stamp (a UTC day at or after the new
    revision's publication) and confirming it resolves before using it.
  • veraPDF: already pinned (v1.30.2), but the tag was duplicated across the image pull and
    the shim it backs. Both now read one job-level VERAPDF_TAG.
  • zxing-cpp: bumped 3.0.03.1.1, with pillow newly pinned to 12.3.0. The 3.0.0 pin
    was recorded against an EAN add-on text-format difference; the affected test
    (EanBarcode_Ean13WithAddOn_MainDigitsExact_AddOnTolerant) only ever asserted the main 13
    digits and treats the add-on's presentation as version-dependent by design, so it passes
    unchanged. Ran the full barcode oracle suite locally against 3.1.1 with
    REQUIRE_BARCODE_ORACLE=1: 1051/1051 passed, no failures. setup-python's interpreter is
    pinned to 3.14 rather than left floating too, for the same interpreter-determinism reason as
    the pip versions beside it — not because of a wheel-availability cliff at 3.15, which doesn't
    exist (zxing-cpp 3.1.1's selected wheel is cp312-abi3, requires_python >=3.9, no upper
    bound).
  • Action bumps: actions/setup-dotnet@v5@v6 (across ci.yml, release.yml,
    docs.yml) and actions/setup-python@v6/@v5@v7 (ci.yml, and docs-inventory.yml,
    which the first pass missed). These ride along in a pinning PR because the whole point of this
    issue is deliberate versions instead of environment drift, and a stale action major is the same
    kind of drift as a stale apt package — worth clearing out at the same time rather than leaving
    for a separate PR to rediscover. dotnet-version: inputs are untouched, as agreed, to keep this
    conflict-free against a separate SDK-pinning PR (merged as ci: pin global.json and CI to the same SDK band #379 while this PR was open —
    global.json now pins the SDK band and every setup-dotnet step reads global-json-file;
    merged main into this branch, no conflicts, @v6 and global-json-file both intact on all
    five).
  • Version-assert step: reads every pinned version from one job-level env: block instead of
    re-hardcoding it, so bumping a pin means editing one line. Checks full-line or full-value
    equality rather than substring greps ("12.4.1" is a substring of "12.4.10"), and asserts the
    apt packages' complete Ubuntu revision via dpkg-query alongside a pdftotext -v identity
    check — dpkg-query proves the package record (pdftotext -v can't see the Ubuntu revision
    after the dash, and fonts can't self-report a version at all), but not which binary the tests
    actually invoke, which is a real ambiguity on Windows per CLAUDE.md.
  • Docs: CONTRIBUTING.md's prerequisites record the pinned versions, the qpdf install method,
    and the poppler/font bump procedure; CLAUDE.md's external-oracles section does the same, so
    "set REQUIRE_ORACLES=1 to reproduce CI locally" has actual versions to match instead of an
    aspiration. CLAUDE.md itself is untracked (.git/info/exclude) — its edit is applied directly
    to the maintainer's working copy and can't appear in this PR's diff.
  • tests/VellumPdf.Reader.Tests/Fixtures/Encrypted/README.md recorded CI as apt/11.9.0 and local
    as 12.3.2; both are now false, so it's updated to 12.4.1 both places. Its two qpdf-12.3.2-specific
    behavioral claims (qpdf pulls the catalog out of an object stream the moment the same run also
    encrypts; qpdf ignores both /Length entries at /V 4 /CFM /V2 and always derives a 16-byte
    key) were re-run against 12.4.1 and both still hold — recorded inline, along with the
    --allow-weak-crypto flag the RC4 half of the first one needs against 12.4.1 (exit 2 without
    it: "refusing to write a file with weak crypto").
  • tests/VellumPdf.TestSupport/OracleGate.cs cited a ci.yml line range for the Test step's env
    block; the range moved once already in this PR, so the comment now names the step instead of a
    line number.

Verification

  • dotnet build VellumPdf.slnx -c Release — 0 warnings, 0 errors.
  • dotnet test VellumPdf.slnx -c Release — all green (oracle tests requiring qpdf/poppler skip
    locally as expected without REQUIRE_ORACLES; ran the barcode suite separately under
    REQUIRE_BARCODE_ORACLE=1 against zxing-cpp 3.1.1, see above).
  • dotnet format VellumPdf.slnx --verify-no-changes — clean.
  • pwsh ./eng/clean-room-check.ps1 — clean.
  • pwsh ./eng/aot/run-aot-smoke.ps1 — passed.
  • Verified the qpdf install step end-to-end in a clean ubuntu:24.04 container (download,
    sha256 check, extract, PATH, qpdf --version reports 12.4.1) before adding it to the
    workflow, and re-verified after adding --retry/unzip -o.
  • Verified the poppler/fonts snapshot install end-to-end in a clean ubuntu:24.04 container:
    the success path (pinned versions install correctly, guard passes silently), an
    already-rotated-off revision (9.1) resolving correctly with live sources still present, and
    both snapshot-unreachable failure modes (TLS trust failure, blackholed host) — confirmed the
    guard fires (exit 1) in both.
  • Confirmed the qpdf 12.4.1 download URL and asset name against
    gh release view v12.4.1 -R qpdf/qpdf --json assets.

Closes #230

apt's qpdf on the runner is 11.9.0, two majors behind upstream, and
--check's output has changed across qpdf majors while #186's
acceptance criterion is exactly that output. Install qpdf 12.4.1 from
the official release artifact (checksum-verified) instead; pin
poppler-utils, fonts-dejavu-core and fonts-texgyre to the exact
versions the runner's apt archive serves today, since an unpinned font
update silently changes fixture rendering for every downstream oracle.

Also deduplicate the veraPDF Docker tag into one job-level env var,
bump zxing-cpp to 3.1.1 with pillow pinned alongside it (the barcode
oracle suite passes unchanged), bump setup-dotnet/setup-python to
their current majors, and add a step that asserts each pinned tool's
own version report so drift fails loudly instead of quietly changing
what CI validates against.

Closes #230
@Tim81 Tim81 added this to the v2.3 — Reader robustness milestone Aug 31, 2026
Tim81 added 4 commits August 31, 2026 09:16
runs-on moves from the floating ubuntu-latest to ubuntu-24.04, since every
apt pin in this job is noble-only and would all break at once the day
GitHub retargets the label.

The poppler/font =version pins alone were a scheduled outage: noble's
-updates/-security pockets keep only the newest revision, so the exact
revisions here would 404 the moment Ubuntu ships another one. apt-get now
passes -o APT::Snapshot for the date those versions were measured, which
rewrites the runner's existing sources to the dated Ubuntu snapshot and
keeps the pinned revisions resolvable indefinitely; verified in a
container with the live archive already past the pinned revision.

setup-python's interpreter is pinned to 3.14 (zxing-cpp 3.1.1 has no
wheel past it) instead of floating, and docs-inventory.yml's own
setup-python, missed by the previous sweep, moves to v7 to match.

The version-assert step now reads every pinned version from job-level env
instead of re-hardcoding them, asserts the apt packages' full Ubuntu
revision via dpkg-query (pdftotext -v can't see it and fonts can't
self-report at all), and compares full lines/values instead of substrings
so a future point release can't slip past a careless grep.

Restored the zxing-cpp pin's inline rationale in ci.yml, which the
previous edit dropped, and updated the CHANGELOG/CONTRIBUTING/CLAUDE.md
prose to match the runner label, the snapshot mechanism and the full apt
version.

tests/VellumPdf.Reader.Tests/Fixtures/Encrypted/README.md's CI/local qpdf
version claim was stale in both directions; both are 12.4.1 now. Reran
its two qpdf-12.3.2-specific behavioral claims (catalog-in-object-stream
under encryption, and /V4 /CFM /V2 ignoring /Length) against 12.4.1 and
recorded that both still hold.

Refs #230
apt-get update exits 0 even when the -o APT::Snapshot fetch itself fails
outright, and silently carries on with whatever else is already cached or
configured. Reproduced both failure shapes in a container (a TLS trust
failure from missing ca-certificates, and a blackholed host) and confirmed
apt gives no other signal in either case, so the poppler/fonts install
step now greps /var/lib/apt/lists/ for the snapshot's own index files
right after the update and fails loudly if they never landed.

The mechanism comment, CONTRIBUTING's paragraph, and the qpdf-1ubuntu9.1
verification claim all overstated or misdescribed how -o APT::Snapshot
works: it adds the dated snapshot alongside the runner's existing
sources rather than rewriting them, ubuntu-24.04 resolves its own
sources through mirror+file:/etc/apt/apt-mirrors.txt rather than a
literal archive.ubuntu.com host, and the earlier "verified against the
live archive already past this revision" claim was false since 9.9 is
still current there today. Re-verified instead against poppler-utils
24.02.0-1ubuntu9.1, a revision already absent from the live archive:
a snapshot dated to when 9.1 was current resolves and installs it
correctly with the live sources still present. The bump-procedure
guidance is corrected too: apt-cache policy cannot produce an
APT_SNAPSHOT date on its own, so it now documents picking a UTC day at
or after the new revision's publication and confirming the stamp
resolves before using it.

Also: setup-python's 3.14 pin was justified by a wheel-availability
cliff at 3.15 that does not exist (zxing-cpp 3.1.1's wheel is
cp312-abi3 with no upper Python bound) — reworded to interpreter
determinism, the same reason the pip versions beside it are pinned.
Restored a pdftotext -v identity check alongside the dpkg-query asserts,
since the package record proves the installed version but not which
binary the tests actually invoke. Fixed a stale ci.yml line-number
citation in OracleGate.cs, a stale ubuntu-latest reference in an
unrelated CHANGELOG entry sitting in the same Unreleased section, and
noted the --allow-weak-crypto flag the RC4 recipe in
Fixtures/Encrypted/README.md needs against qpdf 12.4.1. aot-smoke's own
ubuntu-latest is noted as deliberate, since it carries no apt pins.

Merged main (PR #379: global.json pins the SDK band, all setup-dotnet
steps read global-json-file) — no conflicts.

Refs #230
The real runner failed the assert step at exit 141 (128+SIGPIPE) right
after the dpkg-query lines: `<tool> ... | head -n1` under `set -euo
pipefail` lets head exit the moment it has its one line, and the
producer (writing a multi-line banner) can still be mid-write when that
happens, so the write hits a closed pipe. Local containers never showed
it because the timing has to line up, which the runner's own buffering
apparently does more often than a clean container does.

Fixed by capturing the full output first and taking the first line with
a bash parameter expansion instead of a second process in the pipeline:
`out=$(cmd); line=${out%%$'\n'*}`. Applied to all three version-banner
reads (qpdf, pdftotext, veraPDF). The snapshot-verification guard had
the same shape with `grep -q` instead of `head` (`ls ... | grep -q ...`)
and got the same treatment: capture the listing, then grep it through a
here-string, which reads a fixed string rather than a live pipe and so
cannot race a still-writing producer.

Re-verified in containers: qpdf/poppler assert logic against a real
install (exit 0), the veraPDF line extraction against real multi-line
`verapdf --version` output, and the guard in both its failing and
passing shapes.

Refs #230
@Tim81
Tim81 merged commit 34ea20a into main Aug 31, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: qpdf and poppler oracles are installed unpinned, so CI is not reproducible

1 participant