Repository navigation
Conversation
The test jobs already compile the same tree with MNE_WARNINGS_AS_ERRORS, so the separate min/max Qt build jobs only repeat that work. Min-Qt tests now run on Linux only. A PR goes from 15 to 6 jobs and from 5 macOS jobs to 1, which matters because the mne-tools runner pool is shared and only 5 macOS jobs can run at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Over the last four weeks the mne-tools runner pool was quietest from about 01:00 to 04:00 UTC, while MNE-Python traffic picks up from 04:00-05:00 UTC and staging pushes usually start around 07:00 UTC. Starting at 01:13 UTC puts the release-build burst after the tests (~1.5 h) inside that window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage is only collected on Linux, but the cache key included the with_code_coverage input on every OS. Staging (coverage on) and PRs (coverage off) therefore saved and looked up different macOS keys for identical builds, so PRs could never hit the macOS cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Okay this seems to work. Can't test |
The runner savings stay; the parts that dropped checks come back where they are cheap or rare: - Staging nightly and manual runs test min Qt on all three OSes again, so every published dev build and the release gate evidence cover each OS x Qt pair. Pushes keep min Qt on Linux only. - The docs gate (API docs, screenshots, site build; Linux, ~20 min) runs on every push again; only the website deployment waits for the nightly. - PRs into main (releases) test min Qt on all three OSes. - CodeQL keeps the reduced triggers but analyses a real build again, now compiled in parallel, so Qt-generated code and template instantiations are covered (build-mode none skips them).
|
Thanks Eric, this is a real problem and the analysis is spot on. I'm happy to take the runner savings. The one thing I don't want to give up is test or analysis coverage, because MNE-CPP is in the middle of a release that is gated on it (every test running on each OS x Qt pair, coverage, docs, CodeQL). So I pushed one commit on top that keeps your savings and puts back only the checks that were dropped, where they are cheap or rare. What stays exactly as you proposed:
What I changed, and why:
Net effect on the shared pool: a staging push goes from ~29 jobs / ~20 runner-hours to 7 jobs (your 5 plus the two Linux docs jobs) and about 5.5 runner-hours. The macOS load per push is the same as in your version. The nightly is 2 jobs larger. PRs into Long jobs also never pile up in the queue: every pipeline cancels its superseded runs. A new push to If the nightly macOS/Windows min-Qt jobs ever collide with other repos at that hour, I'm happy to move or thin them. I'd just like to keep the coverage itself. |
…the build changes
Every push runs the quality and docs gates and all six test configurations,
so each failure and coverage change is reported on the commit that caused
it. The release builds, installers, WebAssembly and deployments also run on
pushes that change CMakeLists.txt, cmake/, scripts/{build,ci,deploy}/ or
.github/workflows/ (any doubt, e.g. a force push, builds), besides nightly
and manual runs.
|
One more adjustment after thinking about how we iterate on
I checked the path detection against recent pushes: our usual library/test pushes come out as tests only, and pushes that touch a CMakeLists.txt or a workflow come out as release. |
|
I had Claude check and it said:
Does that make sense as a way to limit things a bit more? |
…filter A push no longer builds releases in its own run, which the next push could cancel after dev_build was deleted. If staging compiles and the build or packaging changed since the last dev build, the push starts a release-only manual run instead; later pushes do not cancel it, and it skips the tests the push already ran. The filter ignores src/testframes/ and doc/ and only counts the staging and release workflows, which the release builds do not depend on otherwise: over the last four weeks it would have matched 39 of 106 pushes instead of 74. The compile gate is a job of its own, shared by both paths.
|
Good points, thanks. I checked the last four weeks and get the same numbers as you: my filter would have started the release builds on 74 of 106 pushes, yours on 39. I pushed a third commit (d695259) with both changes:
|
|
Okay for me! |
WebAssembly breaks in ways no other build sees (32 sources have WebAssembly-only code paths), so it can't wait for a release run. The build moves to a reusable workflow that every staging push and every PR runs (one Linux job); staging publishes it to dev_build and the demo site only in release runs. scripts/wasm/ now also counts as a packaging change.
|
One more commit (2d027e2): WebAssembly now builds on every staging push and on PRs. It breaks in ways no other build sees (32 source files have WebAssembly-only code paths), so it shouldn't wait for a release run. It's one Linux job of about 40 minutes and no macOS. Publishing to |
|
Okay looks green 👍 |
Reduce CI usage on staging while keeping full test coverage - Every staging push runs the quality and docs gates, all tests on Linux, macOS and Windows with minimum and maximum Qt (with coverage), and the WebAssembly build. Superseded push runs are cancelled. - Release builds, installers and deployments run nightly (when staging moved), on manual dispatch, and after a push that changed the build or packaging since the last dev build, as a separate run that later pushes do not cancel. They start only once every test configuration compiles. A failing nightly opens an issue that closes when a later nightly passes. - The separate BuildMinQt/BuildMaxQt jobs are dropped from staging and PRs: the test jobs compile the same tree with warnings as errors. - PRs run tests on all three OSes (max Qt), min Qt on Linux (all three OSes for PRs into main) and the WebAssembly build. - CodeQL runs weekly, on main, on PRs into main and on manual dispatch, on a full parallel build. - Builds compile in parallel; Linux/macOS test builds use a size-capped ccache that only staging saves.
|
Merged into |
Closes #990 (again, hopefully for good... it's not actually open but it would close it I think).
Claude Opus 5.5 summary
The 20 GitHub-hosted runners (5 macOS) are shared by every repo in
mne-tools. Since late Septemberstaginghas received 6–8 pushes a day, and each push started the full release pipeline: 29 jobs and roughly 20 runner-hours (1036–1283 runner-minutes across the last 8 runs), most of them starting at once and 7 of them on macOS. On top of that, CodeQL ran on every push, took ~4–4.5 h, and was usually cancelled by the next push before finishing. In practice MNE-CPP kept most of the org's pool busy all day, and other repos (MNE-Python, MNE-BIDS-Pipeline, themne-toolsadmin workflows, …) sat queued for hours.Changes
Staging pipeline (
staging.yml)stagingmoved since the last nightly) and on manual dispatch (Actions → Staging → Run workflow).dev_buildis only replaced once every test configuration compiles. Failing tests don't block dev builds.BuildMinQt/BuildMaxQtare dropped from staging: Tests and TestsMinQt already compile the same tree withMNE_WARNINGS_AS_ERRORS, and the nightly release builds cover deployment.Pull request pipeline (
pull-request.yml)BuildMinQt/BuildMaxQt(the test jobs already compile withMNE_WARNINGS_AS_ERRORS) and run min-Qt tests on Linux only: 15 → 6 jobs per PR, 5 → 1 macOS jobs. Packaging is still exercised by the nightly release builds and bymain.yml, which is unchanged.CodeQL (
codeql.yml)staging. It still runs weekly, onmain, on PRs tomain, and now on manual dispatch.build-mode: none, so there's no Qt/Eigen download or full compile, which should take it from ~4 h to minutes.Build speed
_reusable-build.ymlnow builds with--parallel. Previously Linux/macOS compiled on a single core._reusable-tests.ymluses ccache on Linux/macOS:mainruns only restore it._reusable-tests.ymltakes a newos_jsoninput to limit which OSes run.Docs: the CI pages under
doc/website/docs/development/now describe the nightly and PR pipelines.Expected effect
Per push to
staging: ~20 → ~5 runner-hours, and ~4–5 jobs starting at once instead of ~20, with ccache cutting further on incremental changes. Per PR: 15 → 6 jobs. Plus one runner freed from CodeQL nearly all day. The full pipeline still runs once a day.Notes / not tested yet
workflow_dispatchand failure issue only take effect once this is onstaging(the default branch). It's worth a manual dispatch after merging to check the release path.staginghas saved a cache, so this PR's own runs compile from scratch.test_com_rt_clientfailed) is correctly allowed.actionlintpasses on the changed workflows, apart from the existing warnings about the undeclaredGH_PATsecret.TL;DR: Each push to
stagingnow runs only the tests: about 5 runner-hours instead of 20, and 5 jobs at once instead of about 20. Dev builds and the docs deploy move to a nightly run that publishes as long as the code compiles. A failing nightly opens an issue that closes itself once a later nightly passes. CodeQL now runs weekly and skips the 4-hour build. Builds now compile in parallel and use a compiler cache with a size cap. PRs drop duplicate build jobs (15 → 6 jobs, 5 → 1 macOS). Day-to-day workflow (pushing tostaging) doesn't change.Opus 5.5 diagnosed usage issues and drafted changes. I didn't see a LLM policy so let me know if you need more from me on this front.