From 2df1230ade71c923daebdb64db4378ddf7387020 Mon Sep 17 00:00:00 2001 From: amazloumi Date: Wed, 26 Aug 2026 13:25:02 -0400 Subject: [PATCH 1/6] Run the examples/ unit test tier in CI The unit-tests job invokes only pytest tests/unit/, and testpaths = ["tests"] means no default collection reaches examples/, so the relocated VLM-eval unit tier ran nowhere. Add an explicit step for it; each tier gets its own pytest session because the tier's conftest injects a fake lmms_eval that would leak into a shared one. The integration tier needs the real, undeclared lmms-eval and stays manual. --- .github/workflows/ci.yml | 5 +++++ CHANGELOG.md | 1 + CONTRIBUTING.md | 2 +- 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5c5522a..3dded39 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,6 +26,11 @@ jobs: - uses: astral-sh/setup-uv@v6 - run: uv sync --group video - run: uv run pytest tests/unit/ -v --timeout=60 --cov --cov-branch --cov-report=xml + # Example tests are outside the root `testpaths`, so each tier is named here + # explicitly. Each needs its own pytest session: the VLM-eval unit conftest + # injects a fake `lmms_eval` that would leak into a tier sharing the session. + # The matching integration tier needs the real, undeclared lmms-eval; manual only. + - run: uv run pytest examples/vlm/eval/tests/unit -v --timeout=60 - name: Upload coverage reports to Codecov uses: codecov/codecov-action@v5 with: diff --git a/CHANGELOG.md b/CHANGELOG.md index fe83c0e..6cfa29b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -107,6 +107,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Training entry point is now a library.** `scripts/train.py:main()` (~1000 lines) is decomposed into `kempnerforge/training/` modules: `runtime.py` (`RuntimeContext`, `PipelineBundle`, `setup_distributed`), `data_pipeline.py` (`DataPipeline`, `PhaseState`, data/eval/phase builders), `loop.py` (`BatchStream`, `StepResult`, `TrainingSession`, `run_training_loop`, and the `text_step` / `vlm_step` / `pipeline_step` bodies picked by `select_step_fn`), and `entry.py` (`run_training` plus the model/checkpoint/resume builders). `scripts/train.py` is now a thin CLI wrapper — same CLI, same log lines, same metric keys, same checkpoint format. Behavior-preserving; the loop is unit-testable with a fake `CheckpointManager` (`tests/unit/test_train_entry.py`). `run_training(config, *, step_fn=None, hooks=None)` lets an experiment own the step body or register hooks without copying the build phases, and both `run_training` and `run_training_loop` tear down in `finally` now that they are library calls rather than a script about to exit. - **BREAKING — VLM evaluation moved out of core into `examples/vlm/eval/`.** `kempnerforge/eval/` and `scripts/vlm_eval_harness.py` are gone; the lmms-eval adapter, CLI, tests and how-to now live in the example, so core carries no lmms-eval-facing code. The `[project.entry-points."lmms_eval.models"]` declaration is removed with no shim — `lmms_eval --model kempnerforge_vlm` no longer resolves, and the harness builds the adapter itself. See `examples/vlm/eval/README.md` for usage. +- CI `unit-tests` job also runs the hermetic example unit tier (`examples/vlm/eval/tests/unit`), which sits outside the root `testpaths`. Each tier gets its own pytest session; the integration tier stays manual (it needs the undeclared `lmms-eval`). - `docs/getting-started/install.md` Prerequisites: documents `.python-version` and uv's auto-fetch behavior. - `README.md` and `kempnerforge/README.md` Prerequisites: clarify that uv auto-fetches Python 3.12 via `.python-version`. - `docs/claude-ready.md` first-run flow: `/kempnerforge:install-and-verify` runs before `/kempnerforge:cluster-config`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 403a1d0..8b64f6a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -200,7 +200,7 @@ CI runs on every push to `main` and every PR. All jobs must pass before merge. | Job | What it checks | Runs on | |-----|----------------|---------| | `lint` | `ruff check` + `ruff format --check` + `pyright` | Every push/PR | -| `unit-tests` | `pytest tests/unit/ -v --timeout=60` | Every push/PR | +| `unit-tests` | `pytest tests/unit/` plus each hermetic `examples/` test tier, one pytest session each | Every push/PR | | `gpu-tests` | `pytest tests/integration/` | Manual dispatch | The most common CI failure is `ruff format --check`. Run `uv run ruff format --check kempnerforge/ tests/ scripts/` locally before pushing. From 05768977daa439bf24128dc3c2883669f8a84dcb Mon Sep 17 00:00:00 2001 From: amazloumi Date: Wed, 26 Aug 2026 14:58:07 -0400 Subject: [PATCH 2/6] Move the example test tier to a path-filtered workflow Run it from its own ci-example.yml on the paths that can affect it rather than on every PR. kempnerforge/** is in the filter because the examples import core (the VLM-eval adapter alone has 10 core imports), so a core-only change can break an example tier without touching examples/ -- the same reason docs.yml filters on kempnerforge/**. --- .github/workflows/ci-example.yml | 29 +++++++++++++++++++++++++++++ .github/workflows/ci.yml | 5 ----- 2 files changed, 29 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/ci-example.yml diff --git a/.github/workflows/ci-example.yml b/.github/workflows/ci-example.yml new file mode 100644 index 0000000..f4c466b --- /dev/null +++ b/.github/workflows/ci-example.yml @@ -0,0 +1,29 @@ +name: CI (examples) + +on: + push: + branches: [main] + pull_request: + branches: [main] + paths: + # `kempnerforge/**` because the examples import core, so a core-only change + # can break an example tier without touching anything under `examples/`. + - "examples/**" + - "kempnerforge/**" + - "pyproject.toml" + - "uv.lock" + - ".github/workflows/ci-example.yml" + workflow_dispatch: + +jobs: + example-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + - uses: astral-sh/setup-uv@v6 + - run: uv sync --group video + # Example tests are outside the root `testpaths`, so each tier is named here + # explicitly. Each needs its own pytest session: the VLM-eval unit conftest + # injects a fake `lmms_eval` that would leak into a tier sharing the session. + # The matching integration tier needs the real, undeclared lmms-eval; manual only. + - run: uv run pytest examples/vlm/eval/tests/unit -v --timeout=60 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3dded39..5c5522a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,11 +26,6 @@ jobs: - uses: astral-sh/setup-uv@v6 - run: uv sync --group video - run: uv run pytest tests/unit/ -v --timeout=60 --cov --cov-branch --cov-report=xml - # Example tests are outside the root `testpaths`, so each tier is named here - # explicitly. Each needs its own pytest session: the VLM-eval unit conftest - # injects a fake `lmms_eval` that would leak into a tier sharing the session. - # The matching integration tier needs the real, undeclared lmms-eval; manual only. - - run: uv run pytest examples/vlm/eval/tests/unit -v --timeout=60 - name: Upload coverage reports to Codecov uses: codecov/codecov-action@v5 with: From bc3a9ccaf1eaf720422a9005fe9c5ff9d73f3982 Mon Sep 17 00:00:00 2001 From: amazloumi Date: Wed, 26 Aug 2026 15:02:00 -0400 Subject: [PATCH 3/6] Narrow the example workflow's PR filter to examples/ and its deps Listing kempnerforge/** made the filter nearly a no-op: 15 of the last 20 merged PRs touched core, so the workflow would have run on 18 of 20 PRs instead of 6. The unfiltered push-to-main trigger covers a core change that breaks an example tier instead, at one merge of latency. --- .github/workflows/ci-example.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci-example.yml b/.github/workflows/ci-example.yml index f4c466b..e5cb35e 100644 --- a/.github/workflows/ci-example.yml +++ b/.github/workflows/ci-example.yml @@ -5,11 +5,11 @@ on: branches: [main] pull_request: branches: [main] + # Scoped to `examples/` and its deps so the filter actually filters. Core is + # deliberately not listed: the unfiltered `push` above is what catches a core + # change that breaks an example tier without touching `examples/`. paths: - # `kempnerforge/**` because the examples import core, so a core-only change - # can break an example tier without touching anything under `examples/`. - "examples/**" - - "kempnerforge/**" - "pyproject.toml" - "uv.lock" - ".github/workflows/ci-example.yml" From 0aa7fff24b489c9fec7a570342720701a4cc29f4 Mon Sep 17 00:00:00 2001 From: amazloumi Date: Wed, 26 Aug 2026 15:02:15 -0400 Subject: [PATCH 4/6] Sync the CI docs with the example workflow Restore the unit-tests row and describe example-tests as its own job. --- CHANGELOG.md | 2 +- CONTRIBUTING.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2a7cd30..287e78d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -108,7 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **BREAKING: the COCO-Karpathy prep helper moved to `examples/vlm/data/prep_vlm_coco.py`** (was `scripts/prep_vlm_coco.py`). It is VLM-experiment data prep, not general tooling, so it belongs with the example rather than in `scripts/`. No compat shim — invoke it at the new path. `--caption-json` and `--image-root` are now required: their old defaults were cluster-specific absolute paths, which a shipped example must not carry. - **Training entry point is now a library.** `scripts/train.py:main()` (~1000 lines) is decomposed into `kempnerforge/training/` modules: `runtime.py` (`RuntimeContext`, `PipelineBundle`, `setup_distributed`), `data_pipeline.py` (`DataPipeline`, `PhaseState`, data/eval/phase builders), `loop.py` (`BatchStream`, `StepResult`, `TrainingSession`, `run_training_loop`, and the `text_step` / `vlm_step` / `pipeline_step` bodies picked by `select_step_fn`), and `entry.py` (`run_training` plus the model/checkpoint/resume builders). `scripts/train.py` is now a thin CLI wrapper — same CLI, same log lines, same metric keys, same checkpoint format. Behavior-preserving; the loop is unit-testable with a fake `CheckpointManager` (`tests/unit/test_train_entry.py`). `run_training(config, *, step_fn=None, hooks=None)` lets an experiment own the step body or register hooks without copying the build phases, and both `run_training` and `run_training_loop` tear down in `finally` now that they are library calls rather than a script about to exit. - **BREAKING — VLM evaluation moved out of core into `examples/vlm/eval/`.** `kempnerforge/eval/` and `scripts/vlm_eval_harness.py` are gone; the lmms-eval adapter, CLI, tests and how-to now live in the example, so core carries no lmms-eval-facing code. The `[project.entry-points."lmms_eval.models"]` declaration is removed with no shim — `lmms_eval --model kempnerforge_vlm` no longer resolves, and the harness builds the adapter itself. See `examples/vlm/eval/README.md` for usage. -- CI `unit-tests` job also runs the hermetic example unit tier (`examples/vlm/eval/tests/unit`), which sits outside the root `testpaths`. Each tier gets its own pytest session; the integration tier stays manual (it needs the undeclared `lmms-eval`). +- New `ci-example.yml` workflow runs the hermetic example test tiers, which sit outside the root `testpaths` and so were collected by no CI step. One pytest session per tier; PR-filtered to `examples/` and its deps, unfiltered on pushes to `main`. The VLM-eval integration tier stays manual (it needs the undeclared `lmms-eval`). - `docs/getting-started/install.md` Prerequisites: documents `.python-version` and uv's auto-fetch behavior. - `README.md` and `kempnerforge/README.md` Prerequisites: clarify that uv auto-fetches Python 3.12 via `.python-version`. - `docs/claude-ready.md` first-run flow: `/kempnerforge:install-and-verify` runs before `/kempnerforge:cluster-config`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8b64f6a..9be686b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -200,8 +200,9 @@ CI runs on every push to `main` and every PR. All jobs must pass before merge. | Job | What it checks | Runs on | |-----|----------------|---------| | `lint` | `ruff check` + `ruff format --check` + `pyright` | Every push/PR | -| `unit-tests` | `pytest tests/unit/` plus each hermetic `examples/` test tier, one pytest session each | Every push/PR | +| `unit-tests` | `pytest tests/unit/ -v --timeout=60` | Every push/PR | | `gpu-tests` | `pytest tests/integration/` | Manual dispatch | +| `example-tests` | each hermetic `examples/` test tier, one pytest session per tier | Push to `main`; PRs touching `examples/` | The most common CI failure is `ruff format --check`. Run `uv run ruff format --check kempnerforge/ tests/ scripts/` locally before pushing. From b25ce48ab7304c865d863ebbb73acacf3d99665e Mon Sep 17 00:00:00 2001 From: amazloumi Date: Wed, 26 Aug 2026 15:46:30 -0400 Subject: [PATCH 5/6] Gate the example tests per example One job per example behind a dorny/paths-filter gate, so touching one example does not run another's tests. Both triggers are path-filtered and the workflow no longer lists its own path, which was matching on every workflow edit and so would have run every example's tests whenever one was added. --- .github/workflows/ci-example.yml | 33 ++++++++++++++++++++++++-------- CHANGELOG.md | 2 +- CONTRIBUTING.md | 2 +- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci-example.yml b/.github/workflows/ci-example.yml index e5cb35e..9fa4a7a 100644 --- a/.github/workflows/ci-example.yml +++ b/.github/workflows/ci-example.yml @@ -3,20 +3,37 @@ name: CI (examples) on: push: branches: [main] + paths: + - "examples/vlm/**" pull_request: branches: [main] - # Scoped to `examples/` and its deps so the filter actually filters. Core is - # deliberately not listed: the unfiltered `push` above is what catches a core - # change that breaks an example tier without touching `examples/`. paths: - - "examples/**" - - "pyproject.toml" - - "uv.lock" - - ".github/workflows/ci-example.yml" + - "examples/vlm/**" workflow_dispatch: +permissions: + contents: read + pull-requests: read + jobs: - example-tests: + # Per-example gate, so touching one example does not run another's tests. Each + # new example adds one output + filter here and one job below. + changes: + runs-on: ubuntu-latest + outputs: + vlm: ${{ steps.filter.outputs.vlm }} + steps: + - uses: actions/checkout@v5 + - uses: dorny/paths-filter@v3 + id: filter + with: + filters: | + vlm: + - "examples/vlm/**" + + vlm-tests: + needs: changes + if: needs.changes.outputs.vlm == 'true' || github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 diff --git a/CHANGELOG.md b/CHANGELOG.md index 287e78d..311d8bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -108,7 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **BREAKING: the COCO-Karpathy prep helper moved to `examples/vlm/data/prep_vlm_coco.py`** (was `scripts/prep_vlm_coco.py`). It is VLM-experiment data prep, not general tooling, so it belongs with the example rather than in `scripts/`. No compat shim — invoke it at the new path. `--caption-json` and `--image-root` are now required: their old defaults were cluster-specific absolute paths, which a shipped example must not carry. - **Training entry point is now a library.** `scripts/train.py:main()` (~1000 lines) is decomposed into `kempnerforge/training/` modules: `runtime.py` (`RuntimeContext`, `PipelineBundle`, `setup_distributed`), `data_pipeline.py` (`DataPipeline`, `PhaseState`, data/eval/phase builders), `loop.py` (`BatchStream`, `StepResult`, `TrainingSession`, `run_training_loop`, and the `text_step` / `vlm_step` / `pipeline_step` bodies picked by `select_step_fn`), and `entry.py` (`run_training` plus the model/checkpoint/resume builders). `scripts/train.py` is now a thin CLI wrapper — same CLI, same log lines, same metric keys, same checkpoint format. Behavior-preserving; the loop is unit-testable with a fake `CheckpointManager` (`tests/unit/test_train_entry.py`). `run_training(config, *, step_fn=None, hooks=None)` lets an experiment own the step body or register hooks without copying the build phases, and both `run_training` and `run_training_loop` tear down in `finally` now that they are library calls rather than a script about to exit. - **BREAKING — VLM evaluation moved out of core into `examples/vlm/eval/`.** `kempnerforge/eval/` and `scripts/vlm_eval_harness.py` are gone; the lmms-eval adapter, CLI, tests and how-to now live in the example, so core carries no lmms-eval-facing code. The `[project.entry-points."lmms_eval.models"]` declaration is removed with no shim — `lmms_eval --model kempnerforge_vlm` no longer resolves, and the harness builds the adapter itself. See `examples/vlm/eval/README.md` for usage. -- New `ci-example.yml` workflow runs the hermetic example test tiers, which sit outside the root `testpaths` and so were collected by no CI step. One pytest session per tier; PR-filtered to `examples/` and its deps, unfiltered on pushes to `main`. The VLM-eval integration tier stays manual (it needs the undeclared `lmms-eval`). +- New `ci-example.yml` workflow runs the hermetic example test tiers, which sit outside the root `testpaths` and so were collected by no CI step. One job per example behind a per-example path filter, so touching one example does not run another's tests; one pytest session per tier. The VLM-eval integration tier stays manual (it needs the undeclared `lmms-eval`). - `docs/getting-started/install.md` Prerequisites: documents `.python-version` and uv's auto-fetch behavior. - `README.md` and `kempnerforge/README.md` Prerequisites: clarify that uv auto-fetches Python 3.12 via `.python-version`. - `docs/claude-ready.md` first-run flow: `/kempnerforge:install-and-verify` runs before `/kempnerforge:cluster-config`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9be686b..6364b31 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -202,7 +202,7 @@ CI runs on every push to `main` and every PR. All jobs must pass before merge. | `lint` | `ruff check` + `ruff format --check` + `pyright` | Every push/PR | | `unit-tests` | `pytest tests/unit/ -v --timeout=60` | Every push/PR | | `gpu-tests` | `pytest tests/integration/` | Manual dispatch | -| `example-tests` | each hermetic `examples/` test tier, one pytest session per tier | Push to `main`; PRs touching `examples/` | +| `-tests` | that example's hermetic test tiers, one pytest session per tier | Pushes/PRs touching `examples//` | The most common CI failure is `ruff format --check`. Run `uv run ruff format --check kempnerforge/ tests/ scripts/` locally before pushing. From 2438705c8fd26ff7f705080743274ae170da5ca9 Mon Sep 17 00:00:00 2001 From: amazloumi Date: Wed, 26 Aug 2026 15:47:14 -0400 Subject: [PATCH 6/6] Note in the example README that CI runs the unit tier --- examples/vlm/eval/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/examples/vlm/eval/README.md b/examples/vlm/eval/README.md index 84e521d..d3dc0b8 100644 --- a/examples/vlm/eval/README.md +++ b/examples/vlm/eval/README.md @@ -202,6 +202,9 @@ uv run pytest examples/vlm/eval/tests/unit uv run pytest examples/vlm/eval/tests/integration ``` +CI runs the unit tier on pushes and PRs that touch `examples/vlm/`; the +integration tier stays manual, since real lmms-eval is not installed there. + Keep the two directories in separate pytest sessions: in a combined run the unit conftest's injected fake replaces the `adapter` module in `sys.modules` after the integration modules have bound the real one, so the monkeypatch-based integration