Skip to content

feat: add multi-image latency benchmarking#729

Open
sjarvie wants to merge 1 commit into
vllm-project:mainfrom
sjarvie:worktree-multi-image-benchmark
Open

feat: add multi-image latency benchmarking#729
sjarvie wants to merge 1 commit into
vllm-project:mainfrom
sjarvie:worktree-multi-image-benchmark

Conversation

@sjarvie

@sjarvie sjarvie commented May 13, 2026

Copy link
Copy Markdown

Summary

Add benchmarking capability to measure latency impact of multiple images (frames) per request in GuideLLM. This enables testing how TTFT and ITL scale with multi-frame vision inputs.

  • Schema: New MultiImageDatasetConfig extending synthetic data generation with images_per_request parameter
  • Generator: 720p synthetic image generation with base64 encoding and reproducible seeding
  • CLI: New --images-per-request parameter supporting single or comma-separated values (e.g., 1,2,5)
  • API: MultiImageBenchmark programmatic class for pytest integration

Test Plan

  • 6 unit tests for config validation (bounds, inheritance, defaults)
  • 8 unit tests for image generator (encoding, reproducibility, dimension accuracy, pixel/byte tracking)
  • All 14 tests passing with proper assertions on JPEG format and base64 validity

Implementation Details

Files Created:

  • src/guidellm/data/schemas.py — MultiImageDatasetConfig
  • src/guidellm/data/generators/multi_image.py — Image generator (720p)
  • src/guidellm/data/generators/__init__.py — Module exports
  • src/guidellm/benchmark/multi_image.py — Programmatic API
  • tests/unit/data/test_multi_image_config.py — Config validation tests
  • tests/unit/data/generators/test_multi_image_generator.py — Generator tests

Files Modified:

  • src/guidellm/data/deserializers/synthetic.py — MultiImageDatasetConfig support
  • src/guidellm/cli/benchmark/run.py — CLI parameter
  • src/guidellm/benchmark/__init__.py — API exports
  • docs/getting-started/benchmark.md — Usage guide with examples

Usage Examples

CLI (single image count):

guidellm benchmark --target http://localhost:8000 --images-per-request 2 --profile constant --rate 10

CLI (multi-variant):

guidellm benchmark --target http://localhost:8000 --images-per-request 1,2,5 --profile constant --rate 10 --max-seconds 30

Programmatic:

from guidellm.benchmark import MultiImageBenchmark

bench = MultiImageBenchmark(image_counts=[1, 2, 5], prompt_tokens=256)
configs = bench.get_configs()  # {1: config, 2: config, 5: config}

@mergify

mergify Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sjarvie.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 21, 2026
Implement multi-image benchmarking for vision-language models to measure
latency impact of multiple frames per request.

Changes:
- MultiImageDatasetConfig schema for datasets with N images per request
- 720p image generator with base64 encoding and reproducible seeding
- CLI parameter: --images-per-request (single or comma-separated list)
- MultiImageBenchmark programmatic API for pytest integration
- 14 unit tests covering config validation and image generation
- Documentation with usage examples

The feature enables benchmarking how TTFT and ITL scale with increasing
frame counts, useful for video analysis pipelines.
@sjarvie sjarvie force-pushed the worktree-multi-image-benchmark branch from 0290221 to 2788605 Compare May 21, 2026 22:21
@mergify mergify Bot removed the needs-rebase label May 21, 2026
@mergify

mergify Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @sjarvie.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label May 27, 2026
@sjmonson sjmonson self-requested a review May 27, 2026 15:26
@mergify

mergify Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Hi @sjarvie, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant