Repository navigation
Make test suite more efficient to run in 10 minutes #584
Description
Description of the desired feature
PyGMT's test suite is growing, and in the long-term, we'll need to make testing a bit more efficient for contributors. As a goal, we should aim for ~10 minutes for the whole test suite (Windows takes up ~30min 15min now (see #586)), and <1 minute for local tests on just one module.
Here's a few ways we can tackle this (note to maintainers, feel free to edit this top post):
Local tests
Running make test can be slow since it runs everything, but sometimes we only want to test one function (e.g. grdimage).
- E.g. at Wrap meca #516 (comment), we should document that contributors can run just one test using something like
pytest -v --mpl pygmt/tests/test_grdimage.py. - Alternatively, look into something like
pytest-fastestto automate testing only the code that has changed between the current branch and master.
Github Actions tests
Installing dependencies
The bulk of the CI time is spent installing dependencies from conda (4min on Linux, 9min on Windows). We could:
- Cache dependencies. This was experimented before, but it didn't seem to help (see Run Continuous Integration tests on Github Actions #475 (comment)), could try again to see things have changed
- Try switching to
mambawhich is a fast reimplementation ofcondawritten in C++. I've tried it before, but it's still in experimental mode at setup-miniconda, see https://github.com/conda-incubator/setup-miniconda#example-6-mamba (Use mamba to install Continuous Integration dependencies #841)
Speed up full test suite
Find a way to run a minimal set of tests when Pull Request is in draft, and only full test suite when Pull Request is ready to review/in review?
Are you willing to help implement and maintain this feature? Let's do it together 😄
Activity
willschlitzer commentedon Jan 4, 2021
@weiji14 Would there be a way to just have the test suite run on one machine (presumably Ubuntu, since that typically is completed in the shortest period of time) in one version of Python when the PR is a draft and then run the full suite when it is ready for review?
Also, would reducing the test suite have any impact on how long it takes for the vercel preview deployment, or is that handled separately?
weiji14 commentedon Jan 4, 2021
Yes it's possible, but you'll need a bit of Github Actions CI experience. Have a look at the GMT Latest tests .yaml file for inspiration. And yes, running just Ubuntu Python 3.9 in draft mode is the idea.
Vercel documentation build is separate, speeding that up would require more work, either from optimizing our Gallery/Tutorial examples or some form of caching on the Python/Sphinx or Javascript/Vercel side. But you could find a way to measure/benchmark it first to see where things run slowest.
willschlitzer commentedon Feb 12, 2021
@weiji14 I'm trying to make a draft test suite, but I'm getting my branch remote rejected because of the OAuth app. It looks like I should be able to generate a personal access token in Developer Settings, but PyGMT isn't listed as an application when I go there to get a token. Any idea how I should go about fixing this?
weiji14 commentedon Feb 12, 2021
Try setting your remote using ssh (
git@github.com:GenericMappingTools/pygmt.git) instead of https (https://github.com/GenericMappingTools/pygmt.git) and see if it works.willschlitzer commentedon Feb 12, 2021
I was able to change the remote but received:
weiji14 commentedon Feb 12, 2021
You also need to add your ssh key to your GitHub account https://docs.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account.
willschlitzer commentedon Feb 12, 2021
It works! Thanks @weiji14!
seisman commentedon Mar 18, 2024
I think we can close the issue since we already achieved our initial goal by a series of PRs. Now the Tests workflow takes 2 minutes on Linux and <8 minutes on Windows.
I think we won't use
pytest-fastest, mainly because:Instead, we may use a similar alternative tracked in #2791.
Closing the issue now.