[MNT] Reduce number of jobs on CI

[amotl]

About

@WilliamJudge94 said:

The main issue is that every push to a PR triggers 20+ CI jobs, which does exceed sktime’s 20 simultaneous runners by itself. This is already causing CI jobs across the organization to sit queued. Especially when multiple PRs are being worked on at the same time.

Proposal

FWIW, this patch evaluates reducing the number of CI jobs.

Review

What do you think, @WilliamJudge94, @siddharth7113, and @fkiraly? Does this improve the situation a bit, and is the patch otherwise acceptable?

[amotl]

in busy times, the total time to wait for the job to finish was reduced from 9h-13h to ~4.75h. Do you agree with this improvement at the cost of less job separation?

[fkiraly]

Obvious question: why does this retain coverage?

There could be different sets of soft deps...

[amotl]

Why does this retain coverage? There could be different sets of soft deps.

I think the patch doesn't do anything about coverage or dependencies. It doesn't touch code at all, but only reorganizes the CI job configuration to emit less jobs. Please educate me if you think I am missing something, or lacking knowledge.

[amotl]

Is this patch like you intended to proceed here, @WilliamJudge94?

[WilliamJudge94]

Is this patch like you intended to proceed here, @WilliamJudge94?

It’s not exactly how I envisioned the implementation based on my previous comments and step list, but the changes you’re proposing here seem quite reasonable. Unfortunately, I don’t currently have access to a laptop, so I’m not able to review the PR as thoroughly as I’d like.

Once I do, one of the main things I’d want to verify is:

1. Are the installation and setup steps for the previously separate CI runs equivalent?

If so, then consolidating them as proposed makes sense to me and I would approve the PR.

If the setup or installation paths were previously different, though, I’d want to understand why before making them uniform. A passing CI alone wouldn’t necessarily prove the change is equivalent—we could inadvertently remove coverage for a configuration that doesn’t currently have a test capable of exposing the regression. Alternatively, the new setup may simply be broader and correctly cover both cases, but I think that distinction is worth verifying.

[amotl]

Hi. Thanks for your swift reply.

Once I do, one of the main things I’d want to verify is: Are the installation and setup steps for the previously separate CI runs equivalent? If so, then consolidating them as proposed makes sense to me and I would approve the PR.

After another self-review, I think I can acknowledge this positively.

It is also worth verifying why now-merged jobs have been distinct before.

I think because many of them are long-running like ~30 minutes each, it was intended to invoke them separately. Of course, there can be plenty of other reasons why separating them was a good idea. What do you think, @fkiraly and @siddharth7113?

[fkiraly]

Are the installation and setup steps for the previously separate CI runs equivalent?

Yes, I would also have that question - that is probably the main issue in whether this merge reatins coverage or not.

[amotl]

The installation instructions for testing notebooks are:

python -m pip install --upgrade pip
python -m pip install uv
python -m uv pip install -U pytest pytest-xdist nbmake

# https://github.com/yaml/pyyaml/issues/736
echo 'Cython < 3.0' > /tmp/constraint.txt
PIP_BUILD_CONSTRAINT=/tmp/constraint.txt pip wheel PyYAML==5.4.1

python -m uv pip install ".[dev,test,full]"
python -m ipykernel install --user --name cikernel

The installation instructions for conducting full, plot, benchmark and tuning tests are:

export SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True
python -m pip install -U pip
python -m pip install -U uv
python -m uv pip install -U pytest numpy

# https://github.com/yaml/pyyaml/issues/736
echo 'Cython < 3.0' > /tmp/constraint.txt
PIP_BUILD_CONSTRAINT=/tmp/constraint.txt pip wheel PyYAML==5.4.1

python -m uv pip install ".[dev,test,full]"
if [ -f requirements-prophet.txt ]; then pip install -r requirements-prophet.txt; fi

[fkiraly]

hm, not exactly the same, is it?

[amotl]

Right, there are two families: Within the same family, setup instructions are the same.

1. Family: Notebooks (with and w/o regression notebook)
2. Family: Full and plot, benchmark and tuning

Only jobs with equal bootstrapping procedures have been grouped and folded into each other, decreasing from 6 individual jobs to 3. Because each job spawned its own matrix across Python versions, this reduction is significant.

[amotl]

Orthogonal to this patch, we submitted another one to trim the bootstrapping surface of CI jobs.

• [MNT] Remove PyYAML and Cython CI workaround #140

[amotl]

Total runtime here can be as fast as 1h50m, while a whole run currently takes 2h50m and more, like 3h25m.