[MNT] restructure CI: conditional PR jobs, move full runs to weekly test-all workflow

[siddharth7113]

Reference Issues/PRs

Follows the CI design of sktime (conditional PR jobs + scheduled full runs).

What does this implement/fix? Explain your changes.

Restructures CI into a lean PR workflow and a weekly full workflow:

• New test_all.yml: runs the weekly cron (Sun 05:00 UTC) and workflow_dispatch, runs the full job set including benchmarks. Scheduled runs are only on sktime/pycaret.

• test.yml (PR CI): cron is removed, test-benchmarks are moved to weekly-only, and a new detect job diffs the PR against its merge base and skips test jobs when no relevant files changed (e.g. docs-only PRs run only code-quality). Skipped jobs still report a status, so required checks stay satisfied. Pushes to main always run everything.

• New test-core job: installs pycaret with core dependencies only (no [full]) and checks that all modules import and the test suite passes without soft dependencies. Supporting test changes: pytest.importorskip guards in 7 test files that imported soft dependencies (mlflow, fugue, boto3/moto) at module level, and a skip for the catboost iterative imputer cases when catboost is not installed.

Does your contribution introduce a new dependency? If yes, which one?

No.

[siddharth7113]

@fkiraly probably a good idea to get this reviewed in priority, I think the current CI workflow is consuming a lot of action minutes as well as unecessary slow

[amotl]

Thank you for this cleanup. I'd tend to run simpler CI/GHA setups, but if it's the way sktime favours it, please go ahead. To detect any hiccups, I am humbly asking @fkiraly to validate this. Thanks!

[amotl]

Are we ready to switch to ruff and ty in any future iteration?

[siddharth7113]

I was thinking the same, I think ty is still in beta (I might be wrong) , but yes switching to ruff is a good idea.

[siddharth7113]

Thank you for this cleanup. I'd tend to run simpler CI/GHA setups, but if it's the way sktime favours it, please go ahead. To detect any hiccups, I am humbly asking @fkiraly to validate this. Thanks!

Yep, I am also new to this 😅, and @fkiraly is much well versed in CI matrix , but my attempt idea is to make sure we aren't running complete CI everytime there is a small change in documentation or parts of code that don't intersect.

[fkiraly]

Questions:

• this seems to delete the tests for the benchmark marker in test without a replacement, except in test_all. That also seems to be the only major coverage change. You mention this in the PR description, but why exactly those? They do not seem to be the major runtime sink?
• the importorskip statements do not isolate imports further down at module level, correct? So, would the test file not fail at test collection, if, say, mlflow is not present?

[siddharth7113]

Questions:

• this seems to delete the tests for the benchmark marker in test without a replacement, except in test_all. That also seems to be the only major coverage change. You mention this in the PR description, but why exactly those? They do not seem to be the major runtime sink?

My assumption with this was benchmarking is something we would check only once a week and not always? Is this something we wish to check at every run time.

[WilliamJudge94]

I think the PyCaret Team should make avoiding regressions the top priority. Tests should run close to when code is pushed so issues are caught while the change is still fresh. If a regression is found a week later, the developer has likely moved on and additional code may already depend on it.

My suggestions for reducing GitHub Actions minutes would be:

1. Docs-only changes: Use paths-ignore within the GitHub Actions workflow so CI does not run when only documentation files are changed.

2. On push: Run the full test suite on only the latest supported Python version.

3. Before merge: Have a maintainer trigger the remaining supported Python versions as a required final check.

This keeps regression feedback fast while reducing unnecessary CI usage.

[amotl]

I agree with @WilliamJudge94. This patch adds ~500 lines to the CI setup, and might lead to regressions. Maybe we should just NOT do it?

My suggestions for reducing GitHub Actions minutes would be [...]

My question is: Do they have to be reduced, or are we chasing ghosts? What is actually the problem?

[siddharth7113]

I agree with @WilliamJudge94. This patch adds ~500 lines to the CI setup, and might lead to regressions. Maybe we should just NOT do it?

My suggestions for reducing GitHub Actions minutes would be [...]

My question is: Do they have to be reduced, or are we chasing ghosts? What is actually the problem?

I think I bundled 2 things together , 1. idea about only relevant parts of CI should run when a change is made, for e.g. when a small readme or documentation is changed, the tests that do not concern them i.e. plotting etc should not run, which in theory might sounds nice, but requires clear boundaries of what should and should not run for parts of code.

2. Reducing some tests to a weekly cron to save CI runtime (which I inherently assumed is needed) .

I think after this discussion I think , it is better to keep the CI as is for now, and revisit at a later point when we have a larger test suite.

[WilliamJudge94]

I agree with @WilliamJudge94. This patch adds ~500 lines to the CI setup, and might lead to regressions. Maybe we should just NOT do it?

My suggestions for reducing GitHub Actions minutes would be [...]

My question is: Do they have to be reduced, or are we chasing ghosts? What is actually the problem?

The main issue is that every push to a PR triggers 20+ CI jobs, which does exceed sktime’s 20 simultaneous runners by itself. This is already causing CI jobs across the organization to sit queued. Especially when multiple PRs are being worked on at the same time.

I agree with @siddharth7113 that the CI needs updating to tackle this problem. We only differ on the implementation.

[fkiraly]

reopening as a draft since we may do a variant of this sometime in the future.