Skip to content

MAINT: Reduce CI usage on staging - #1013

Closed
larsoner wants to merge 8 commits into
mne-tools:stagingfrom
larsoner:ci/reduce-runner-usage
Closed

larsoner wants to merge 8 commits into
mne-tools:stagingfrom
larsoner:ci/reduce-runner-usage

Conversation

@larsoner

@larsoner larsoner commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Closes #990 (again, hopefully for good... it's not actually open but it would close it I think).

Claude Opus 5.5 summary

The 20 GitHub-hosted runners (5 macOS) are shared by every repo in mne-tools. Since late September staging has received 6–8 pushes a day, and each push started the full release pipeline: 29 jobs and roughly 20 runner-hours (1036–1283 runner-minutes across the last 8 runs), most of them starting at once and 7 of them on macOS. On top of that, CodeQL ran on every push, took ~4–4.5 h, and was usually cancelled by the next push before finishing. In practice MNE-CPP kept most of the org's pool busy all day, and other repos (MNE-Python, MNE-BIDS-Pipeline, the mne-tools admin workflows, …) sat queued for hours.

Changes

Staging pipeline (staging.yml)

  • On push: only Quality, Tests (3 OSes, with coverage) and TestsMinQt (now Linux only) run, so 5 test jobs instead of 29.
  • Dev release binaries, installers, WebAssembly, Docs/DocShots, Doxygen, website and release evidence now run nightly (only if staging moved since the last nightly) and on manual dispatch (Actions → Staging → Run workflow).
  • dev_build is only replaced once every test configuration compiles. Failing tests don't block dev builds.
  • A failing nightly opens a "Nightly staging build failed" issue (or comments on the open one), which closes automatically when a later nightly passes.
  • A push no longer cancels a nightly or manual run.
  • BuildMinQt/BuildMaxQt are dropped from staging: Tests and TestsMinQt already compile the same tree with MNE_WARNINGS_AS_ERRORS, and the nightly release builds cover deployment.

Pull request pipeline (pull-request.yml)

  • Drop BuildMinQt/BuildMaxQt (the test jobs already compile with MNE_WARNINGS_AS_ERRORS) and run min-Qt tests on Linux only: 15 → 6 jobs per PR, 5 → 1 macOS jobs. Packaging is still exercised by the nightly release builds and by main.yml, which is unchanged.

CodeQL (codeql.yml)

  • No longer runs on pushes to staging. It still runs weekly, on main, on PRs to main, and now on manual dispatch.
  • Uses build-mode: none, so there's no Qt/Eigen download or full compile, which should take it from ~4 h to minutes.

Build speed

  • _reusable-build.yml now builds with --parallel. Previously Linux/macOS compiled on a single core.
  • _reusable-tests.yml uses ccache on Linux/macOS:
    • One cache entry per OS, Qt version and coverage setting, capped at 400 MB each, so about 1.2 GB in total.
    • Only staging saves the cache, and it deletes the entries it replaces, so the cache doesn't grow and push out the Qt and sample-data caches.
    • PR and main runs only restore it.
  • _reusable-tests.yml takes a new os_json input to limit which OSes run.

Docs: the CI pages under doc/website/docs/development/ now describe the nightly and PR pipelines.

Expected effect

Per push to staging: ~20 → ~5 runner-hours, and ~4–5 jobs starting at once instead of ~20, with ccache cutting further on incremental changes. Per PR: 15 → 6 jobs. Plus one runner freed from CodeQL nearly all day. The full pipeline still runs once a day.

Notes / not tested yet

  • The nightly schedule, workflow_dispatch and failure issue only take effect once this is on staging (the default branch). It's worth a manual dispatch after merging to check the release path.
  • ccache only pays off once staging has saved a cache, so this PR's own runs compile from scratch.
  • I tested the compile-gate logic against two past runs: run 38271708364 (Windows link error) is correctly blocked, and run 38358916890 (everything compiled, test_com_rt_client failed) is correctly allowed.
  • actionlint passes on the changed workflows, apart from the existing warnings about the undeclared GH_PAT secret.

TL;DR: Each push to staging now runs only the tests: about 5 runner-hours instead of 20, and 5 jobs at once instead of about 20. Dev builds and the docs deploy move to a nightly run that publishes as long as the code compiles. A failing nightly opens an issue that closes itself once a later nightly passes. CodeQL now runs weekly and skips the 4-hour build. Builds now compile in parallel and use a compiler cache with a size cap. PRs drop duplicate build jobs (15 → 6 jobs, 5 → 1 macOS). Day-to-day workflow (pushing to staging) doesn't change.

Opus 5.5 diagnosed usage issues and drafted changes. I didn't see a LLM policy so let me know if you need more from me on this front.

larsoner and others added 4 commits October 13, 2026 10:28
The test jobs already compile the same tree with MNE_WARNINGS_AS_ERRORS,
so the separate min/max Qt build jobs only repeat that work. Min-Qt tests
now run on Linux only. A PR goes from 15 to 6 jobs and from 5 macOS jobs
to 1, which matters because the mne-tools runner pool is shared and only
5 macOS jobs can run at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Over the last four weeks the mne-tools runner pool was quietest from about
01:00 to 04:00 UTC, while MNE-Python traffic picks up from 04:00-05:00 UTC
and staging pushes usually start around 07:00 UTC. Starting at 01:13 UTC
puts the release-build burst after the tests (~1.5 h) inside that window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage is only collected on Linux, but the cache key included the
with_code_coverage input on every OS. Staging (coverage on) and PRs
(coverage off) therefore saved and looked up different macOS keys for
identical builds, so PRs could never hit the macOS cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@larsoner

Copy link
Copy Markdown
Member Author

Okay this seems to work. Can't test ccache until it hits staging, then it should be even faster, since ~80 min of the Windows build is the build itself for example. Will be cool to see if it actually speeds things up there!

The runner savings stay; the parts that dropped checks come back where they
are cheap or rare:
- Staging nightly and manual runs test min Qt on all three OSes again, so
  every published dev build and the release gate evidence cover each OS x Qt
  pair. Pushes keep min Qt on Linux only.
- The docs gate (API docs, screenshots, site build; Linux, ~20 min) runs on
  every push again; only the website deployment waits for the nightly.
- PRs into main (releases) test min Qt on all three OSes.
- CodeQL keeps the reduced triggers but analyses a real build again, now
  compiled in parallel, so Qt-generated code and template instantiations are
  covered (build-mode none skips them).
@chdinh

chdinh commented Oct 14, 2026

Copy link
Copy Markdown
Member

Thanks Eric, this is a real problem and the analysis is spot on. I'm happy to take the runner savings. The one thing I don't want to give up is test or analysis coverage, because MNE-CPP is in the middle of a release that is gated on it (every test running on each OS x Qt pair, coverage, docs, CodeQL). So I pushed one commit on top that keeps your savings and puts back only the checks that were dropped, where they are cheap or rare.

What stays exactly as you proposed:

  • Pushes to staging run only Quality, Tests (3 OSes, coverage) and min-Qt tests on Linux. Release builds, installers, WebAssembly, Doxygen and the website move to the nightly and manual runs.
  • The compile gate before dev_build (an improvement, since today nothing gates it), the failure issue, and the push/nightly concurrency split.
  • No separate BuildMinQt/BuildMaxQt jobs (the test jobs compile the same tree), parallel builds, and the bounded ccache.
  • CodeQL off staging pushes: weekly, on main, on PRs to main and on manual dispatch.

What I changed, and why:

  1. Min Qt on all three OSes in the nightly and manual runs (pushes stay Linux-only). Without it, min Qt on macOS and Windows is never tested anywhere before a release, and our release gate needs every OS x Qt pair. Cost: about +2.5 runner-hours once a day (one macOS and one Windows job), nothing per push.
  2. The docs gate on every push again (DocShots + Docs, Linux only, about 20 runner-minutes, no macOS). It is the API docs gate and is cheap. Only the website deployment waits for the nightly.
  3. PRs into main test min Qt on all three OSes. These are release PRs, a handful per year.
  4. CodeQL analyses a real build again (build-mode: manual), now with --parallel. build-mode: none doesn't see the Qt-generated code (moc/uic/rcc) or template instantiations and is less accurate. Since it no longer runs per push, the full build costs one Linux runner about once a week plus releases. The 4 h runs came from a single-core compile being cancelled by the next push, and both of those are gone.

Net effect on the shared pool: a staging push goes from ~29 jobs / ~20 runner-hours to 7 jobs (your 5 plus the two Linux docs jobs) and about 5.5 runner-hours. The macOS load per push is the same as in your version. The nightly is 2 jobs larger. PRs into staging are exactly as you proposed.

Long jobs also never pile up in the queue: every pipeline cancels its superseded runs. A new push to staging cancels the running push pipeline, a new commit on a PR cancels that PR's run, and the same holds for main and CodeQL. So a burst of pushes costs only the last run, not one full run each. The one exception is deliberate: a push never cancels a running nightly or manual run, so the daily release pipeline always finishes.

If the nightly macOS/Windows min-Qt jobs ever collide with other repos at that hour, I'm happy to move or thin them. I'd just like to keep the coverage itself.

…the build changes

Every push runs the quality and docs gates and all six test configurations,
so each failure and coverage change is reported on the commit that caused
it. The release builds, installers, WebAssembly and deployments also run on
pushes that change CMakeLists.txt, cmake/, scripts/{build,ci,deploy}/ or
.github/workflows/ (any doubt, e.g. a force push, builds), besides nightly
and manual runs.
@chdinh

chdinh commented Oct 14, 2026

Copy link
Copy Markdown
Member

One more adjustment after thinking about how we iterate on staging: we push often and rely on CI to report on the exact commit that broke something, so I pushed a second commit (5445f29):

  • Every push runs all six test configurations (three OSes, minimum and maximum Qt) plus the quality and docs gates. Compared with my first commit, that adds the macOS and Windows min-Qt jobs per push: about 6.7 instead of 5.5 runner-hours per push, still about a third of today's ~20.
  • The release builds, installers, WebAssembly and deployments also run on pushes that change the build or packaging (CMakeLists.txt, cmake/, scripts/{build,ci,deploy}/, .github/workflows/). Those are the pushes that can break them. Every other push skips them, as in your version; the nightly and manual runs still do the full pipeline. If the push can't be compared (new branch, force push, very large diff), it builds to be safe.

I checked the path detection against recent pushes: our usual library/test pushes come out as tests only, and pushes that touch a CMakeLists.txt or a workflow come out as release.

@larsoner

Copy link
Copy Markdown
Member Author

I had Claude check and it said:

My one concern is the release-on-push filter. You checked the last few pushes, and those are tests-only. But over the last 4 weeks it would have matched 74 of 106 staging pushes, mostly because of src/testframes/**/CMakeLists.txt and _reusable-tests.yml, since pushes bundle ~4 commits. That puts us at ~11 runner-hours per push, and each match runs 5 macOS jobs, the whole org's macOS limit.

Could we exclude test-only paths (src/testframes/, doc/, workflows other than staging.yml/release-*.yml)? That would have matched 39 of 106 pushes. Only top-level build files would match 18 of 106. Either keeps "release builds when the build or packaging changes". Also, a release started by a push can currently be cancelled by the next push after dev_build has been deleted. Maybe use a non-cancelling concurrency group when release == true?

Does that make sense as a way to limit things a bit more?

…filter

A push no longer builds releases in its own run, which the next push could
cancel after dev_build was deleted. If staging compiles and the build or
packaging changed since the last dev build, the push starts a release-only
manual run instead; later pushes do not cancel it, and it skips the tests the
push already ran. The filter ignores src/testframes/ and doc/ and only counts
the staging and release workflows, which the release builds do not depend on
otherwise: over the last four weeks it would have matched 39 of 106 pushes
instead of 74. The compile gate is a job of its own, shared by both paths.
@chdinh

chdinh commented Oct 14, 2026

Copy link
Copy Markdown
Member

Good points, thanks. I checked the last four weeks and get the same numbers as you: my filter would have started the release builds on 74 of 106 pushes, yours on 39. I pushed a third commit (d695259) with both changes:

  • Your narrower filter. Changes to tests (src/testframes/) and docs (doc/) no longer start the release builds, and of the workflows only the staging and release ones count. The release builds don't build the tests, so we lose nothing. I kept the library and app CMakeLists.txt files in, because a new plugin is a common way to break packaging.
  • Releases are no longer cancelled. A push doesn't build releases itself anymore. If something compiles and the build changed since the last dev build, the push starts a separate release run. Later pushes don't cancel it, and it doesn't rerun the tests.

@larsoner

Copy link
Copy Markdown
Member Author

Okay for me!

WebAssembly breaks in ways no other build sees (32 sources have
WebAssembly-only code paths), so it can't wait for a release run. The build
moves to a reusable workflow that every staging push and every PR runs (one
Linux job); staging publishes it to dev_build and the demo site only in
release runs. scripts/wasm/ now also counts as a packaging change.
@chdinh

chdinh commented Oct 15, 2026

Copy link
Copy Markdown
Member

One more commit (2d027e2): WebAssembly now builds on every staging push and on PRs. It breaks in ways no other build sees (32 source files have WebAssembly-only code paths), so it shouldn't wait for a release run. It's one Linux job of about 40 minutes and no macOS. Publishing to dev_build and the demo site still only happens in release runs.

@larsoner

Copy link
Copy Markdown
Member Author

Okay looks green 👍

chdinh pushed a commit that referenced this pull request Oct 15, 2026
Reduce CI usage on staging while keeping full test coverage

- Every staging push runs the quality and docs gates, all tests on Linux,
  macOS and Windows with minimum and maximum Qt (with coverage), and the
  WebAssembly build. Superseded push runs are cancelled.
- Release builds, installers and deployments run nightly (when staging moved),
  on manual dispatch, and after a push that changed the build or packaging
  since the last dev build, as a separate run that later pushes do not cancel.
  They start only once every test configuration compiles. A failing nightly
  opens an issue that closes when a later nightly passes.
- The separate BuildMinQt/BuildMaxQt jobs are dropped from staging and PRs:
  the test jobs compile the same tree with warnings as errors.
- PRs run tests on all three OSes (max Qt), min Qt on Linux (all three OSes
  for PRs into main) and the WebAssembly build.
- CodeQL runs weekly, on main, on PRs into main and on manual dispatch, on a
  full parallel build.
- Builds compile in parallel; Linux/macOS test builds use a size-capped
  ccache that only staging saves.
@chdinh

chdinh commented Oct 15, 2026

Copy link
Copy Markdown
Member

Merged into staging as a single squash commit, eb850e8. Thanks Eric!

@chdinh chdinh closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GitHub actions runners resource usage

2 participants