What
VLMs are currently single-process single-GPU evaluated via lmms-eval. lmms-eval has built in support for multi-GPU eval via data parallelism that we can integrate, allowing us to speed up eval time.
Scope
Limited to eval/vlm/adapter.py and scripts/vlm_eval_harness.py.
Backward compatibility
This will preserve all current behavior.
What
VLMs are currently single-process single-GPU evaluated via
lmms-eval.lmms-evalhas built in support for multi-GPU eval via data parallelism that we can integrate, allowing us to speed up eval time.Scope
Limited to
eval/vlm/adapter.pyandscripts/vlm_eval_harness.py.Backward compatibility
This will preserve all current behavior.