Run the examples/ unit test tier in CI

[amazloumi]

Summary

• New ci-example.yml runs the hermetic example test tiers, which sit outside the root testpaths and so were collected by no CI step after Move VLM evaluation into examples/vlm/eval #184. One job per example behind a dorny/paths-filter gate, so touching one example does not run another's tests. Today that is vlm-tests running pytest examples/vlm/eval/tests/unit; a second tier in the same example is one more - run: line.
• Both triggers are path-filtered to examples/vlm/**. The workflow's own path is deliberately not listed: it matched on every edit to the file, so adding an example would have run every example's tests.
• One pytest session per tier — the tier's conftest injects a hermetic fake lmms_eval that would leak into any tier sharing the session. The VLM-eval integration tier needs the real, undeclared lmms-eval and stays manual.
• ci.yml is byte-identical to main; ruff and pyright scope unchanged.
• Adds dorny/paths-filter@v3 — the first action used here outside actions/, astral-sh/ and codecov/.
• Consequence: with both triggers filtered, a core change that breaks an example tier is not caught by this workflow; it surfaces when the tier next runs. The VLM-eval adapter has 10 imports from kempnerforge.
• Consequence: the ruleset's required contexts are lint and unit-tests. A path-filtered check cannot be made required — GitHub blocks every PR where it never reports — so these jobs are advisory unless someone adds them to the ruleset and accepts that.
• Open question, not settled here: pyproject.toml and uv.lock are not in the filter, so a dependency bump can break an example tier without touching any example directory. "Run every tier" is a different rule from per-example filtering, so it seems worth deciding separately.

Testing

• uv run pytest tests/unit/ -v --timeout=120 on the final tree (2438705) — 1726 passed, 3 skipped in 237.76s
• The tier in a venv proven free of lmms-eval — find_spec("lmms_eval") is None, import lmms_eval raises ModuleNotFoundError, and no orphaned lmms_eval/ directory is on sys.path. find_spec alone is not sufficient: uv sync prunes the package's modules but leaves its task-data files, and that __init__.py-less directory reads as a namespace package. Result: 104 passed, exit 0.
• On the runner, changes resolved dorny/paths-filter@v3 and logged Filter vlm = true / Changes output set to ["vlm"]; the gated vlm-tests job then reported collected 104 items / 104 passed in 6.78s.
• Skip case observed, which the earlier filter could not show: b25ce48 touched only .github/workflows/ci-example.yml, CHANGELOG.md and CONTRIBUTING.md — CI ran and CI (examples) produced no run at all. 2438705 then added examples/vlm/eval/README.md and CI (examples) ran and passed.
• The job can fail: reversing the comparison in adapter.py::_first_stop, so it returns the last stop match instead of the first, gives 1 failed, 103 passed and exit 1. Defect reverted.
• Test path is examples/vlm/eval/tests/unit, which exists on this base. examples/vlm/tests (added by Move VLM training configs into examples/vlm/ #193) does not, and naming it would exit 4 — confirmed by running it.
• uv run ruff check kempnerforge/ tests/ scripts/ — All checks passed. uv run ruff format --check kempnerforge/ tests/ scripts/ — 166 files already formatted.
• uv run pyright kempnerforge/ — 0 errors, 0 warnings, 0 informations.
• ci-example.yml parses to jobs changes + vlm-tests, with permissions: contents/pull-requests: read; ci.yml is byte-identical to main.
• Distributed / fault-tolerance / resumability battery deliberately not run: the diff is workflow YAML plus docs and touches no runtime code, so there is no path by which it can affect those guarantees.

Closes #191

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Naeemkh]

Thank you.