Skip to content

[Benchmark] Add Support for LVOmniBench#1510

Open
KD-TAO wants to merge 1 commit intoopen-compass:mainfrom
KD-TAO:lvomnibench
Open

[Benchmark] Add Support for LVOmniBench#1510
KD-TAO wants to merge 1 commit intoopen-compass:mainfrom
KD-TAO:lvomnibench

Conversation

@KD-TAO
Copy link
Copy Markdown

@KD-TAO KD-TAO commented Apr 9, 2026

Summary

Add LVOmniBench, a benchmark for evaluating omnimodal LLMs on long-form audio-video understanding.

Evaluation

  • Fine-grained accuracy across 4 dimensions:
    • question_type (9 categories): Attribute Perception, Counting, Event Understanding, etc.
    • audio_type (3 categories): Speech, Music, Sound
    • difficulty (3 levels): High, Medium, Low
    • video_category (6 categories): Film & TV, Entertainment, Sports, etc.

Changes

File Change
vlmeval/dataset/lvomnibench.py New dataset class
vlmeval/dataset/utils/lvomnibench.py Evaluation utilities
vlmeval/dataset/__init__.py Registration (+2 lines)
vlmeval/dataset/video_dataset_config.py Config variants (+11 lines)

Test

# Default (video_llm mode, no frame extraction)              
python run.py --data LVOmniBench --model Qwen2.5-Omni-7B-ForVideo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant