Skip to content

CI: Various caching fixes #717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
notroj wants to merge 9 commits into from
Closed

CI: Various caching fixes #717

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d496678
CI: Share one CPAN cache across the matrix.
notroj Aug 26, 2026
53c8de5
CI: Key the ~/root cache on the dependencies, retiring CLEAR_CACHE.
notroj Aug 26, 2026
51027fd
CI: Actually set MFLAGS, so builds are not serial.
notroj Aug 26, 2026
4294cd8
CI: Drop the subversion package from the OpenSSL jobs.
notroj Aug 26, 2026
c9eb34f
CI: Only install cpanminus, perl-doc and gdb where they are used.
notroj Aug 26, 2026
50f74a1
CI: Split the ASan jobs so the test suites run in parallel.
notroj Aug 26, 2026
1b25e09
CI: Limit job concurrency as the ASF policy requires.
notroj Aug 26, 2026
5000073
CI: Start the slowest jobs first.
notroj Aug 27, 2026
97d5882
CI: Move the APLOGNO() check out of the build matrix.
notroj Aug 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 146 additions & 103 deletions .github/workflows/linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ permissions:
contents: read

env:
MARGS: "-j2"
CFLAGS: "-g"
# This will need updating as the ubuntu-latest image changes:
PHP_FPM: "/usr/sbin/php-fpm8.3"
Expand All @@ -39,13 +38,112 @@ jobs:
build:
strategy:
fail-fast: false
# https://infra.apache.org/github-actions-policy.html requires a
# job concurrency level of 20 or less.
max-parallel: 20
matrix:
include:
# -------------------------------------------------------------------------
- name: Empty APLOGNO() test
- name: Shared MPMs (event), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=event
env: |
SKIP_TESTING=1
TEST_LOGNO=1
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
# -------------------------------------------------------------------------
- name: Shared MPMs (worker), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=worker
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
# -------------------------------------------------------------------------
- name: Shared MPMs (prefork), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=prefork
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
# -------------------------------------------------------------------------
- name: UBSan
notest-cflags: -fsanitize=undefined -fno-sanitize-recover=undefined
config: --enable-mods-shared=reallyall --disable-http2
env: |
NOTEST_LIBS=-lubsan
TEST_UBSAN=1
TEST_PYTEST=1
# -------------------------------------------------------------------------
# The ASan jobs are the slowest in the matrix and so set the
# wall-clock time for the whole workflow; running the Perl
# framework and the pytest suites in separate jobs halves that.
# Both halves build the same dependencies, so they share one
# ~/root cache entry.
- name: ASan
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer
config: --enable-mods-shared=reallyall
env: |
APR_VERSION=1.7.x
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
# -------------------------------------------------------------------------
- name: ASan, pytest
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer
config: --enable-mods-shared=reallyall
pkgs: nghttp2-client
env: |
APR_VERSION=1.7.x
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
NO_TEST_FRAMEWORK=1
TEST_PYTEST=1
# -------------------------------------------------------------------------
- name: ASan, pool-debug
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer -O2
config: --enable-mods-shared=reallyall --enable-maintainer-mode
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
# -------------------------------------------------------------------------
- name: ASan, pool-debug, pytest
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer -O2
config: --enable-mods-shared=reallyall --enable-maintainer-mode
pkgs: nghttp2-client
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
NO_TEST_FRAMEWORK=1
TEST_PYTEST=1
# -------------------------------------------------------------------------
# Runs every pytest-based test suite (pytest_suite/ + all
# test/modules/*/ pyhttpd suites except modules/md, which needs a
# local ACME/pebble server that isn't available here) via `make
# check-all-pytest`. See TEST_PYTEST in test/travis_run_linux.sh.
- name: Python pytest test suites
config: --enable-mods-shared=reallyall --with-mpm=event --enable-mpms-shared=all
pkgs: nghttp2-client
env: |
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--with-crypto"
NO_TEST_FRAMEWORK=1
TEST_PYTEST=1
# -------------------------------------------------------------------------
- name: Default
# -------------------------------------------------------------------------
Expand Down Expand Up @@ -117,15 +215,13 @@ jobs:
APR_VERSION=1.7.x
APU_VERSION=1.6.x
APU_CONFIG="--with-crypto --with-ldap"
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: APR 1.8.x, APR-util 1.7.x
config: --enable-mods-shared=reallyall
env: |
APR_VERSION=1.8.x
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: APR 1.7.x, APR-util 1.7.x, LDAP
config: --enable-mods-shared=reallyall
Expand All @@ -139,7 +235,6 @@ jobs:
TEST_ARGS="-defines LDAP"
TESTS="t/modules/"
TEST_PYTEST=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: APR trunk thread debugging
config: --enable-mods-shared=reallyall --with-mpm=event
Expand All @@ -155,40 +250,6 @@ jobs:
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: Shared MPMs (event), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=event
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: Shared MPMs (worker), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=worker
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: Shared MPMs (prefork), pool-debug, SSL/TLS variants
config: --enable-mods-shared=reallyall --enable-mpms-shared=all --with-mpm=prefork
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_MALLOC=1
TEST_SSL=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: litmus WebDAV tests
config: --enable-dav --enable-dav-fs
Expand All @@ -215,53 +276,6 @@ jobs:
BUILDCONFIG="--with-regen-expr"
MFLAGS=
# -------------------------------------------------------------------------
- name: UBSan
notest-cflags: -fsanitize=undefined -fno-sanitize-recover=undefined
config: --enable-mods-shared=reallyall --disable-http2
env: |
NOTEST_LIBS=-lubsan
TEST_UBSAN=1
TEST_PYTEST=1
# -------------------------------------------------------------------------
- name: ASan
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer
config: --enable-mods-shared=reallyall
pkgs: nghttp2-client
env: |
APR_VERSION=1.7.x
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
TEST_PYTEST=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
- name: ASan, pool-debug
notest-cflags: -ggdb -fsanitize=address -fno-sanitize-recover=address -fno-omit-frame-pointer -O2
config: --enable-mods-shared=reallyall --enable-maintainer-mode
pkgs: nghttp2-client
env: |
APR_VERSION=1.7.x
APR_CONFIG="--enable-pool-debug"
APU_VERSION=1.7.x
APU_CONFIG="--with-crypto --with-ldap"
TEST_ASAN=1
TEST_PYTEST=1
CLEAR_CACHE=1
# -------------------------------------------------------------------------
# Runs every pytest-based test suite (pytest_suite/ + all
# test/modules/*/ pyhttpd suites except modules/md, which needs a
# local ACME/pebble server that isn't available here) via `make
# check-all-pytest`. See TEST_PYTEST in test/travis_run_linux.sh.
- name: Python pytest test suites
config: --enable-mods-shared=reallyall --with-mpm=event --enable-mpms-shared=all
pkgs: nghttp2-client
env: |
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--with-crypto"
NO_TEST_FRAMEWORK=1
TEST_PYTEST=1
# -------------------------------------------------------------------------
### TODO: if: *condition_not_24x
- name: Configured w/reduced exports
config: --enable-reduced-exports --enable-maintainer-mode --enable-systemd
Expand Down Expand Up @@ -293,7 +307,6 @@ jobs:
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--without-crypto"
pkgs: subversion
# -------------------------------------------------------------------------
- name: OpenSSL 3.4 -Werror
config: --enable-mods-shared=most --enable-maintainer-mode --disable-md --disable-http2 --disable-ldap --disable-crypto
Expand All @@ -303,7 +316,6 @@ jobs:
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--without-crypto"
pkgs: subversion
# -------------------------------------------------------------------------
- name: OpenSSL 3.4 no-engine
config: --enable-mods-shared=most --enable-maintainer-mode --disable-md --disable-http2 --disable-ldap --disable-crypto
Expand All @@ -313,7 +325,6 @@ jobs:
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--without-crypto"
pkgs: subversion
# -------------------------------------------------------------------------
- name: OpenSSL 3.5 no-engine -Werror
config: --enable-mods-shared=most --enable-maintainer-mode --disable-md --disable-http2 --disable-ldap --disable-crypto
Expand All @@ -324,7 +335,6 @@ jobs:
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--without-crypto"
pkgs: subversion
# -------------------------------------------------------------------------
- name: OpenSSL 4.0
config: --enable-mods-shared=most --enable-maintainer-mode --disable-md --disable-http2 --disable-ldap --disable-crypto
Expand All @@ -335,7 +345,6 @@ jobs:
APR_VERSION=1.7.6
APU_VERSION=1.6.5
APU_CONFIG="--without-crypto"
pkgs: subversion
# -------------------------------------------------------------------------
runs-on: ${{ matrix.os == '' && 'ubuntu-latest' || matrix.os }}
timeout-minutes: 30
Expand All @@ -344,42 +353,76 @@ jobs:
CONFIG: ${{ matrix.config }}
name: ${{ matrix.name }}
steps:
# JOBID is used in the cache keys, created here as a hash of all
# properties of the environment, including the image OS version,
# compiler flags and any job-specific properties.
# JOBID is used to name the artifacts uploaded on failure. IMAGEOS
# is exported so that it is visible to the env context, which does
# not see the runner's own $ImageOS.
- name: Set environment variables
run: |
echo "${{ matrix.env }}" >> $GITHUB_ENV
echo "IMAGEOS=$ImageOS" >> $GITHUB_ENV
# The scripts run "make $MFLAGS"; default it to the number of
# CPUs the runner has, unless the matrix entry set it already
# (the ap_expr job sets it empty on purpose).
grep -q '^MFLAGS=' $GITHUB_ENV || echo "MFLAGS=-j`nproc`" >> $GITHUB_ENV
echo JOBID=`echo "OS=$ImageOS ${{ matrix.notest-cflags }} ${{ matrix.env }} ${{ matrix.config }}" \
| md5sum - | sed 's/ .*//'` >> $GITHUB_ENV
- name: apt refresh
run: sudo apt-get -o Acquire::Retries=5 update
# cpanminus and perl-doc are only used to build the CPAN modules for
# the Perl test framework, and perl-doc drags in around a hundred
# packages. gdb is only used to extract a backtrace from any core
# dumped during testing, which includes the pytest suites.
- name: Install prerequisites
run: sudo apt-get install -o Acquire::Retries=5
cpanminus libtool-bin libapr1-dev libaprutil1-dev
libtool-bin libapr1-dev libaprutil1-dev
liblua5.3-dev libbrotli-dev libcurl4-openssl-dev
libnghttp2-dev libjansson-dev libpcre2-dev gdb
perl-doc libsasl2-dev curl pipx ${{ matrix.pkgs }} check
libnghttp2-dev libjansson-dev libpcre2-dev
${{ ! ( env.SKIP_TESTING || env.NO_TEST_FRAMEWORK ) && 'cpanminus perl-doc' || '' }}
${{ ! env.SKIP_TESTING && 'gdb' || '' }}
libsasl2-dev curl pipx ${{ matrix.pkgs }} check
- uses: actions/checkout@v6
- uses: actions/checkout@v6
with:
repository: apache/httpd-tests
path: test/perl-framework
submodules: true
if: ${{ ! ( env.SKIP_TESTING || env.NO_TEST_FRAMEWORK ) }}
- name: Cache installed libraries
uses: actions/cache@v5
# Sets $ROOTID, identifying the contents of ~/root, and leaves it
# unset for a job which builds no dependencies from source.
- name: Resolve dependency versions
run: ./test/gha-resolve-deps.sh
# Restored here but saved only once the dependencies have been built
# successfully, below: a job which fails part-way through building
# them would otherwise save a half-built tree under a key which can
# never be updated. Bump the serial in the key to discard the cached
# builds and start afresh.
- name: Restore installed libraries
uses: actions/cache/restore@v5
id: cache-libs
if: env.ROOTID != ''
with:
path: ~/root
key: cache-libs-${{ env.JOBID }}
key: cache-libs-1-${{ env.IMAGEOS }}-${{ runner.arch }}-${{ env.ROOTID }}
# ~/perl5 does not vary by job: the module list is fixed and the
# modules are always built with CC=gcc. It varies only by the perl
# ABI (hence the image and architecture) and by the module list
# itself, so one cache is shared by every job. Bump the serial in
# the key to force a rebuild against current CPAN.
- name: Cache CPAN modules
uses: actions/cache@v5
if: ${{ ! ( env.SKIP_TESTING || env.NO_TEST_FRAMEWORK ) }}
with:
path: ~/perl5
key: cache-cpan-${{ env.JOBID }}
key: cache-cpan-1-${{ env.IMAGEOS }}-${{ runner.arch }}-${{ hashFiles('test/travis_before_linux.sh') }}
- name: Configure environment
run: ./test/travis_before_linux.sh
timeout-minutes: 15
- name: Save installed libraries
uses: actions/cache/save@v5
if: success() && env.ROOTID != '' && steps.cache-libs.outputs.cache-hit != 'true'
with:
path: ~/root
key: ${{ steps.cache-libs.outputs.cache-primary-key }}
- uses: actions/upload-artifact@v7
if: failure()
with:
Expand Down
Loading