Feat/text multi image eval

examples/vlm-evaluation/README.md

@@ -119,12 +119,37 @@ uv run python examples/vlm-evaluation/vlm_eval_harness.py \
   on the checkpoint's frame budget matching the reference's, which is a training
   choice rather than a knob here.
 - **Scope.** One video per request, single-turn, zero-shot, generative arches
-  (`joint_decoder` / `cross_attention` / `mot`). A single **image** task also runs
-  on a video checkpoint — the image is treated as a 1-frame clip, zero-padded to
-  `frames_per_clip`. Multiple videos, mixed image+video, multiple images, audio,
-  and multi-turn / few-shot raise a clear error; MoMa still fails fast. An
-  **image** checkpoint cannot evaluate video and raises a clear error if handed a
-  video task.
+  (`joint_decoder` / `cross_attention` / `mot`). **Image** tasks also run on a
+  video checkpoint — one image is a 1-frame clip, and multiple images are packed
+  as an ordered clip (zero-padded, and truncated with a warning past
+  `frames_per_clip`). Multiple videos, mixed image+video, audio, and multi-turn /
+  few-shot raise a clear error; MoMa still fails fast. An **image** checkpoint
+  cannot evaluate video and raises a clear error if handed a video task.
+
+## Text-only evaluation
+
+Text-only `generate_until` benchmarks (e.g. GSM8K, IFEval) run on **both image and
+video checkpoints**, for the generative arches (`joint_decoder` / `cross_attention` /
+`mot`). A request with no image or video renders as an empty-frame prompt and runs
+the arch's **pure-text forward** — no vision encoder, no image prefix (JD/MoT), and
+cross-attention blocks skipped (CA) — so the number reflects the text backbone. This
+is how you measure how much VLM training drifted the base LM, in the same harness as
+the multimodal tasks.
+
+```bash
+uv run python examples/vlm-evaluation/vlm_eval_harness.py \
+    --config     configs/train/vlm_jd.toml \
+    --checkpoint checkpoints/vlm/step_10000 \
+    --tasks      gsm8k \
+    --limit      8
+```
+
+- **Scope.** `generate_until` tasks only (generation / answer-extraction).
+  `loglikelihood`-scored multiple-choice suites (ARC, HellaSwag, MMLU-style) are not
+  supported — the adapter is generation-only. MoMa is excluded (non-generative).
+  Text-only, image, and video requests may be freely mixed across a task suite; each
+  request is decoded by its own modality path (text-only and visual requests within a
+  batch are decoded as separate sub-batches).
 
 ## Limitations
 
@@ -139,13 +164,13 @@ Several are tracked follow-ups.
   generation-only. A MoMa checkpoint fails fast with a clear error. Joint-Decoder
   (`joint_decoder`), Cross-Attention (`cross_attention`), and MoT (`mot`) are
   supported.
-- **One visual per request; no multi-turn / few-shot / multi-image.** A request
-  carries exactly one image (image checkpoint) or one video (video checkpoint —
-  see [Video evaluation](#video-evaluation)). Audio, multiple images, multiple
-  videos, mixed image+video, and multi-turn / few-shot requests raise a clear
-  error. Multi-image and multi-turn/few-shot are tracked follow-ups (for chat
-  tasks lmms-eval delivers few-shot as extra content blocks/turns, so it reduces
-  to multi-image + multi-turn support).
+- **One visual per request on image checkpoints; no multi-turn / few-shot.** An
+  image checkpoint carries exactly one image per request (multiple images raise); a
+  video checkpoint carries one video, or one or more images packed as an ordered
+  clip (see [Video evaluation](#video-evaluation)). Audio, multiple videos, mixed
+  image+video, and multi-turn / few-shot requests raise a clear error. Multi-turn /
+  few-shot is a tracked follow-up (for chat tasks lmms-eval delivers few-shot as
+  extra content blocks/turns, so it reduces to multi-turn support).
 - **Prompt flattening discards structure.** Flattening drops role/turn structure
   and any model-specific chat template. KempnerForge pre-training uses no chat
   template; once a post-training format exists, repo-wide chat-template support

examples/vlm-evaluation/tests/integration/test_lmms_eval_contract.py

@@ -19,8 +19,18 @@
         "fake lmms_eval is active; skipping real-package contract tests", allow_module_level=True
     )
 
-from lmms_eval.api.instance import Instance  # noqa: E402
+from lmms_eval.api.instance import (  # noqa: E402
+    GenerationResult,
+    Instance,
+    TokenCounts,
+    unwrap_generation_output,
+)
 from lmms_eval.api.model import lmms  # noqa: E402
+from lmms_eval.models.model_utils.gen_metrics import (  # noqa: E402
+    log_metrics,
+    reset_logged_metrics,
+    summarize_logged_metrics,
+)
 from lmms_eval.protocol import ChatMessages  # noqa: E402
 from lmms_eval.utils import Collator  # noqa: E402
 
@@ -90,3 +100,34 @@ def test_args_returns_arguments_tuple(self):
             metadata={"task": "t", "doc_id": "d0", "repeats": 1},
         )
         assert inst.args == ("ctx", None, {}, "d0", "t", "test")
+
+
+class TestGenerationResultContract:
+    """Pins the typed generate_until return + per-sample token counters the adapter now emits."""
+
+    def test_generation_result_and_token_counts_fields(self):
+        gr = GenerationResult(text="a b c", token_counts=TokenCounts(output_tokens=3))
+        assert gr.text == "a b c"
+        assert gr.token_counts.output_tokens == 3
+        # to_dict drops None fields — the shape build_efficiency_summary consumes.
+        assert gr.token_counts.to_dict() == {"output_tokens": 3}
+
+    def test_unwrap_generation_output_handles_str_and_wrapper(self):
+        text, tc = unwrap_generation_output(
+            GenerationResult(text="x", token_counts=TokenCounts(output_tokens=1))
+        )
+        assert text == "x" and tc.output_tokens == 1
+        # A bare string (the pre-instrumentation return) unwraps to (text, None).
+        assert unwrap_generation_output("plain") == ("plain", None)
+
+
+class TestGenMetricsContract:
+    """Pins the throughput sink the adapter calls; the evaluator resets/summarizes this."""
+
+    def test_log_metrics_feeds_summary(self):
+        reset_logged_metrics()
+        log_metrics(total_elapsed_time=2.0, total_gen_tokens=10, avg_speed=5.0)
+        summary = summarize_logged_metrics()
+        assert summary["total_gen_tokens"] == 10
+        assert summary["total_elapsed_time"] == 2.0
+        reset_logged_metrics()

examples/vlm-evaluation/tests/integration/test_vlm_eval.py

@@ -36,7 +36,7 @@
     pytest.skip("fake lmms_eval is active; skipping real-package tests", allow_module_level=True)
 
 from adapter import KempnerForgeVLM  # noqa: E402
-from lmms_eval.api.instance import Instance  # noqa: E402
+from lmms_eval.api.instance import GenerationResult, Instance  # noqa: E402
 
 from kempnerforge.config.data import DataConfig  # noqa: E402
 from kempnerforge.config.schema import JobConfig  # noqa: E402
@@ -110,8 +110,9 @@ def doc_to_messages(doc):
 
     outputs = vlm.generate_until(instances)
     assert isinstance(outputs, list) and len(outputs) == 2
-    assert all(isinstance(o, str) for o in outputs)
-    assert all(len(o.split()) == 3 for o in outputs)  # greedy emits exactly max_new_tokens
+    assert all(isinstance(o, GenerationResult) for o in outputs)
+    assert all(len(o.text.split()) == 3 for o in outputs)  # greedy emits exactly max_new_tokens
+    assert all(o.token_counts.output_tokens == 3 for o in outputs)  # per-sample output count
 
 
 def test_dcp_roundtrip_video_generate_until(tmp_path, tiny_video_configs, monkeypatch):
@@ -177,8 +178,9 @@ def doc_to_messages(doc):
 
     outputs = vlm.generate_until(instances)
     assert isinstance(outputs, list) and len(outputs) == 2
-    assert all(isinstance(o, str) for o in outputs)
-    assert all(len(o.split()) == 3 for o in outputs)  # greedy emits exactly max_new_tokens
+    assert all(isinstance(o, GenerationResult) for o in outputs)
+    assert all(len(o.text.split()) == 3 for o in outputs)  # greedy emits exactly max_new_tokens
+    assert all(o.token_counts.output_tokens == 3 for o in outputs)  # per-sample output count
 
 
 @pytest.mark.skipif(

examples/vlm-evaluation/tests/unit/_fake_lmms_eval.py

@@ -18,6 +18,10 @@
 - ``api.model.lmms``: base sets ``_rank=0/_world_size=1/cache_hook/task_dict`` and exposes
   ``rank``/``world_size`` properties.
 - ``api.instance.Instance``: dataclass exposing ``.args`` (the arguments tuple).
+- ``api.instance.GenerationResult`` / ``TokenCounts``: the typed ``generate_until``
+  return and per-request token counters (``.text`` / ``.token_counts.output_tokens``).
+- ``models.model_utils.gen_metrics.log_metrics`` (+ ``reset_logged_metrics`` /
+  ``summarize_logged_metrics``): the throughput history the evaluator resets/summarizes.
 """
 
 from __future__ import annotations
@@ -172,6 +176,74 @@ def args(self) -> tuple:
         return self.arguments if isinstance(self.arguments, tuple) else (self.arguments,)
 
 
+# --------------------------------------------------------------------------- #
+# lmms_eval.api.instance.TokenCounts / GenerationResult
+# --------------------------------------------------------------------------- #
+
+
+@dataclasses.dataclass
+class TokenCounts:
+    input_tokens: int | None = None
+    output_tokens: int | None = None
+    reasoning_tokens: int | None = None
+
+    def to_dict(self) -> dict[str, int | None]:
+        d: dict[str, int | None] = {}
+        if self.input_tokens is not None:
+            d["input_tokens"] = self.input_tokens
+        if self.output_tokens is not None:
+            d["output_tokens"] = self.output_tokens
+        if self.reasoning_tokens is not None:
+            d["reasoning_tokens"] = self.reasoning_tokens
+        return d
+
+
+@dataclasses.dataclass
+class GenerationResult:
+    text: str
+    token_counts: TokenCounts | None = None
+
+
+# --------------------------------------------------------------------------- #
+# lmms_eval.models.model_utils.gen_metrics (throughput history)
+# --------------------------------------------------------------------------- #
+
+_THROUGHPUT_METRICS_HISTORY: list[dict[str, Any]] = []
+
+
+def reset_logged_metrics() -> None:
+    _THROUGHPUT_METRICS_HISTORY.clear()
+
+
+def log_metrics(
+    total_elapsed_time: float,
+    total_gen_tokens: int,
+    avg_speed: float,
+    additional_metrics: dict[str, Any] | None = None,
+) -> None:
+    payload: dict[str, Any] = {
+        "total_elapsed_time": total_elapsed_time,
+        "total_gen_tokens": total_gen_tokens,
+        "avg_speed": avg_speed,
+    }
+    if additional_metrics:
+        payload.update(additional_metrics)
+    _THROUGHPUT_METRICS_HISTORY.append(payload)
+
+
+def summarize_logged_metrics() -> dict[str, Any]:
+    if not _THROUGHPUT_METRICS_HISTORY:
+        return {}
+    total_gen_tokens = sum(m.get("total_gen_tokens", 0) for m in _THROUGHPUT_METRICS_HISTORY)
+    total_elapsed_time = sum(m.get("total_elapsed_time", 0.0) for m in _THROUGHPUT_METRICS_HISTORY)
+    avg_speed = (total_gen_tokens / total_elapsed_time) if total_elapsed_time > 0 else 0.0
+    return {
+        "total_gen_tokens": total_gen_tokens,
+        "total_elapsed_time": total_elapsed_time,
+        "avg_speed": avg_speed,
+    }
+
+
 # --------------------------------------------------------------------------- #
 # Module tree assembly
 # --------------------------------------------------------------------------- #
@@ -193,15 +265,31 @@ def _mod(name: str, **attrs: Any) -> types.ModuleType:
     root = _mod("lmms_eval")
     api = _mod("lmms_eval.api")
     api_model = _mod("lmms_eval.api.model", lmms=lmms, CacheHook=_CacheHook)
-    api_instance = _mod("lmms_eval.api.instance", Instance=Instance)
+    api_instance = _mod(
+        "lmms_eval.api.instance",
+        Instance=Instance,
+        GenerationResult=GenerationResult,
+        TokenCounts=TokenCounts,
+    )
     protocol = _mod("lmms_eval.protocol", ChatMessages=ChatMessages)
     utils = _mod("lmms_eval.utils", Collator=Collator)
+    models = _mod("lmms_eval.models")
+    model_utils = _mod("lmms_eval.models.model_utils")
+    gen_metrics = _mod(
+        "lmms_eval.models.model_utils.gen_metrics",
+        log_metrics=log_metrics,
+        reset_logged_metrics=reset_logged_metrics,
+        summarize_logged_metrics=summarize_logged_metrics,
+    )
 
     root.api = api
     root.protocol = protocol
     root.utils = utils
+    root.models = models
     api.model = api_model
     api.instance = api_instance
+    models.model_utils = model_utils
+    model_utils.gen_metrics = gen_metrics
 
     return {
         "lmms_eval": root,
@@ -210,4 +298,7 @@ def _mod(name: str, **attrs: Any) -> types.ModuleType:
         "lmms_eval.api.instance": api_instance,
         "lmms_eval.protocol": protocol,
         "lmms_eval.utils": utils,
+        "lmms_eval.models": models,
+        "lmms_eval.models.model_utils": model_utils,
+        "lmms_eval.models.model_utils.gen_metrics": gen_metrics,
     }