diff --git a/.github/workflows/ci-example.yml b/.github/workflows/ci-example.yml new file mode 100644 index 0000000..9fa4a7a --- /dev/null +++ b/.github/workflows/ci-example.yml @@ -0,0 +1,46 @@ +name: CI (examples) + +on: + push: + branches: [main] + paths: + - "examples/vlm/**" + pull_request: + branches: [main] + paths: + - "examples/vlm/**" + workflow_dispatch: + +permissions: + contents: read + pull-requests: read + +jobs: + # Per-example gate, so touching one example does not run another's tests. Each + # new example adds one output + filter here and one job below. + changes: + runs-on: ubuntu-latest + outputs: + vlm: ${{ steps.filter.outputs.vlm }} + steps: + - uses: actions/checkout@v5 + - uses: dorny/paths-filter@v3 + id: filter + with: + filters: | + vlm: + - "examples/vlm/**" + + vlm-tests: + needs: changes + if: needs.changes.outputs.vlm == 'true' || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + - uses: astral-sh/setup-uv@v6 + - run: uv sync --group video + # Example tests are outside the root `testpaths`, so each tier is named here + # explicitly. Each needs its own pytest session: the VLM-eval unit conftest + # injects a fake `lmms_eval` that would leak into a tier sharing the session. + # The matching integration tier needs the real, undeclared lmms-eval; manual only. + - run: uv run pytest examples/vlm/eval/tests/unit -v --timeout=60 diff --git a/CHANGELOG.md b/CHANGELOG.md index b57e65c..311d8bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -108,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **BREAKING: the COCO-Karpathy prep helper moved to `examples/vlm/data/prep_vlm_coco.py`** (was `scripts/prep_vlm_coco.py`). It is VLM-experiment data prep, not general tooling, so it belongs with the example rather than in `scripts/`. No compat shim — invoke it at the new path. `--caption-json` and `--image-root` are now required: their old defaults were cluster-specific absolute paths, which a shipped example must not carry. - **Training entry point is now a library.** `scripts/train.py:main()` (~1000 lines) is decomposed into `kempnerforge/training/` modules: `runtime.py` (`RuntimeContext`, `PipelineBundle`, `setup_distributed`), `data_pipeline.py` (`DataPipeline`, `PhaseState`, data/eval/phase builders), `loop.py` (`BatchStream`, `StepResult`, `TrainingSession`, `run_training_loop`, and the `text_step` / `vlm_step` / `pipeline_step` bodies picked by `select_step_fn`), and `entry.py` (`run_training` plus the model/checkpoint/resume builders). `scripts/train.py` is now a thin CLI wrapper — same CLI, same log lines, same metric keys, same checkpoint format. Behavior-preserving; the loop is unit-testable with a fake `CheckpointManager` (`tests/unit/test_train_entry.py`). `run_training(config, *, step_fn=None, hooks=None)` lets an experiment own the step body or register hooks without copying the build phases, and both `run_training` and `run_training_loop` tear down in `finally` now that they are library calls rather than a script about to exit. - **BREAKING — VLM evaluation moved out of core into `examples/vlm/eval/`.** `kempnerforge/eval/` and `scripts/vlm_eval_harness.py` are gone; the lmms-eval adapter, CLI, tests and how-to now live in the example, so core carries no lmms-eval-facing code. The `[project.entry-points."lmms_eval.models"]` declaration is removed with no shim — `lmms_eval --model kempnerforge_vlm` no longer resolves, and the harness builds the adapter itself. See `examples/vlm/eval/README.md` for usage. +- New `ci-example.yml` workflow runs the hermetic example test tiers, which sit outside the root `testpaths` and so were collected by no CI step. One job per example behind a per-example path filter, so touching one example does not run another's tests; one pytest session per tier. The VLM-eval integration tier stays manual (it needs the undeclared `lmms-eval`). - `docs/getting-started/install.md` Prerequisites: documents `.python-version` and uv's auto-fetch behavior. - `README.md` and `kempnerforge/README.md` Prerequisites: clarify that uv auto-fetches Python 3.12 via `.python-version`. - `docs/claude-ready.md` first-run flow: `/kempnerforge:install-and-verify` runs before `/kempnerforge:cluster-config`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 403a1d0..6364b31 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -202,6 +202,7 @@ CI runs on every push to `main` and every PR. All jobs must pass before merge. | `lint` | `ruff check` + `ruff format --check` + `pyright` | Every push/PR | | `unit-tests` | `pytest tests/unit/ -v --timeout=60` | Every push/PR | | `gpu-tests` | `pytest tests/integration/` | Manual dispatch | +| `-tests` | that example's hermetic test tiers, one pytest session per tier | Pushes/PRs touching `examples//` | The most common CI failure is `ruff format --check`. Run `uv run ruff format --check kempnerforge/ tests/ scripts/` locally before pushing. diff --git a/examples/vlm/eval/README.md b/examples/vlm/eval/README.md index 84e521d..d3dc0b8 100644 --- a/examples/vlm/eval/README.md +++ b/examples/vlm/eval/README.md @@ -202,6 +202,9 @@ uv run pytest examples/vlm/eval/tests/unit uv run pytest examples/vlm/eval/tests/integration ``` +CI runs the unit tier on pushes and PRs that touch `examples/vlm/`; the +integration tier stays manual, since real lmms-eval is not installed there. + Keep the two directories in separate pytest sessions: in a combined run the unit conftest's injected fake replaces the `adapter` module in `sys.modules` after the integration modules have bound the real one, so the monkeypatch-based integration