CI: Various caching fixes - #717
Closed
notroj wants to merge 9 commits into
Closed
Conversation
notroj
force-pushed
the
ci-better-caching
branch
3 times, most recently
from
August 26, 2026 22:02
b14cd59 to
d87ccc1
Compare
~/perl5 was keyed on $JOBID, giving one near-identical copy per matrix entry; its content depends only on the perl ABI and the module list. * .github/workflows/linux.yml: Key it on the image, architecture and a hash of travis_before_linux.sh, and skip it for jobs which do not use the Perl framework. * test/README.ci: Describe the cache keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Caches in GHA are immutable, so a key which does not change when a dependency does can never be refreshed; branch builds were therefore rebuilt on every run, and CLEAR_CACHE disabled caching altogether for the nine jobs using them. The key also hashed the whole matrix entry, so jobs building identical dependencies each had their own copy. * test/gha-resolve-deps.sh: Added. Compute $ROOTID from the dependency versions, their configure arguments and the compiler flags, resolving any branch to a commit; export the commit so that the build uses what the key names rather than resolving it a second time. * .github/workflows/linux.yml: Key ~/root on $ROOTID, skip the cache where nothing is built from source, and save it only once the dependencies have built successfully. Drop CLEAR_CACHE=1 from the nine jobs which set it. * test/travis_before_linux.sh (resolve_apx, fetch_apx): Split out of install_apx(). (install_apx): Fetch the APR source before building APR-util if APR came from the cache; trust a restored install root. Drop the CLEAR_CACHE handling and the .key-* and openssl-is-* staleness files, which the cache key now covers. * test/README.ci: Update accordingly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MARGS is read by nothing. * .github/workflows/linux.yml: Drop MARGS. Default MFLAGS to the runner's CPU count, unless the matrix entry sets it. * test/README.ci: Update the MFLAGS description. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Added in r1921311 for the "svn co" of Apache-Test, which went away in r1934474 along with the last use of svn in CI. * .github/workflows/linux.yml: Remove pkgs: subversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cpanminus and perl-doc build the CPAN modules for the Perl test framework, and perl-doc pulls in around a hundred packages; gdb only extracts backtraces from core dumps. * .github/workflows/linux.yml: Install the first two only for jobs running the framework, and gdb only for jobs which run tests at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ASan jobs are the slowest in the matrix and so set the wall-clock time for the whole workflow; testing is three quarters of it. * .github/workflows/linux.yml: Run the Perl framework and the pytest suites in separate jobs for each ASan configuration. Both halves build the same dependencies, so they share a ~/root cache entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
notroj
force-pushed
the
ci-better-caching
branch
from
August 26, 2026 22:18
d87ccc1 to
50f74a1
Compare
The matrix ran all its jobs at once - 37 in the last run - where https://infra.apache.org/github-actions-policy.html allows at most 20. * .github/workflows/linux.yml: Set max-parallel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With max-parallel set, jobs start in matrix order, so the long ones have to start first or they run alone at the end; this is worth two minutes of wall clock. * .github/workflows/linux.yml: Move the SSL/TLS variant, ASan, UBSan and pytest jobs to the head of the matrix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It needs only a checkout: of the 23 seconds it took as a matrix job, one was the check and the rest was installing packages it never used. * .github/workflows/sanity.yml: Added, for quick checks which need nothing built. * .github/workflows/linux.yml: Drop the job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
~/perl5 was keyed on $JOBID, giving one near-identical copy per matrix entry; its content depends only on the perl ABI and the module list. * .github/workflows/linux.yml: Key it on the image, architecture and a hash of travis_before_linux.sh, and skip it for jobs which do not use the Perl framework. * test/README.ci: Describe the cache keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937497 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
Caches in GHA are immutable, so a key which does not change when a dependency does can never be refreshed; branch builds were therefore rebuilt on every run, and CLEAR_CACHE disabled caching altogether for the nine jobs using them. The key also hashed the whole matrix entry, so jobs building identical dependencies each had their own copy. * test/gha-resolve-deps.sh: Added. Compute $ROOTID from the dependency versions, their configure arguments and the compiler flags, resolving any branch to a commit; export the commit so that the build uses what the key names rather than resolving it a second time. * .github/workflows/linux.yml: Key ~/root on $ROOTID, skip the cache where nothing is built from source, and save it only once the dependencies have built successfully. Drop CLEAR_CACHE=1 from the nine jobs which set it. * test/travis_before_linux.sh (resolve_apx, fetch_apx): Split out of install_apx(). (install_apx): Fetch the APR source before building APR-util if APR came from the cache; trust a restored install root. Drop the CLEAR_CACHE handling and the .key-* and openssl-is-* staleness files, which the cache key now covers. * test/README.ci: Update accordingly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937498 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
MARGS is read by nothing. * .github/workflows/linux.yml: Drop MARGS. Default MFLAGS to the runner's CPU count, unless the matrix entry sets it. * test/README.ci: Update the MFLAGS description. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937499 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
Added in r1921311 for the "svn co" of Apache-Test, which went away in r1934474 along with the last use of svn in CI. * .github/workflows/linux.yml: Remove pkgs: subversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937500 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
…ip ci] cpanminus and perl-doc build the CPAN modules for the Perl test framework, and perl-doc pulls in around a hundred packages; gdb only extracts backtraces from core dumps. * .github/workflows/linux.yml: Install the first two only for jobs running the framework, and gdb only for jobs which run tests at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937501 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
The ASan jobs are the slowest in the matrix and so set the wall-clock time for the whole workflow; testing is three quarters of it. * .github/workflows/linux.yml: Run the Perl framework and the pytest suites in separate jobs for each ASan configuration. Both halves build the same dependencies, so they share a ~/root cache entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937502 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
The matrix ran all its jobs at once - 37 in the last run - where https://infra.apache.org/github-actions-policy.html allows at most 20. * .github/workflows/linux.yml: Set max-parallel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937503 13f79535-47bb-0310-9956-ffa450edef68
asf-gitbox-commits
pushed a commit
that referenced
this pull request
Aug 27, 2026
With max-parallel set, jobs start in matrix order, so the long ones have to start first or they run alone at the end; this is worth two minutes of wall clock. * .github/workflows/linux.yml: Move the SSL/TLS variant, ASan, UBSan and pytest jobs to the head of the matrix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> GitHub: PR #717 git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1937504 13f79535-47bb-0310-9956-ffa450edef68
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.