I’d like to suggest adding EvidenceBench, a benchmark I created, to your list of agent benchmarks.
EvidenceBench is a multimodal AI benchmark designed to evaluate whether models can distinguish what is actually supported by visual evidence from what is merely technically plausible.
The initial benchmark, CardinalBenchmark-001 (cb-001), uses a deliberately complex Cardinal (VCV Rack) modular-synthesis patch as a controlled evaluation environment. The goal is not to test whether a model knows how a particular synthesizer module works, but whether it can maintain accurate information about modules, cables, spatial relationships, and technical connections when the visual environment becomes difficult to interpret.
CB-001 uses a three-turn evaluation:
- Visual analysis — models identify and reason about what can be directly established from the supplied visual evidence, including recognizing ambiguity and handling uncertainty.
- Structured connection audit — models compare visual evidence against an engineer-provided JSON connection record and identify incorrect, duplicated, missing, misspelled, or visually unsupported claims. The engineer’s documentation is treated as claims to audit rather than ground truth.
- Evidence reassessment — models receive improved visual evidence and reassess conclusions that were previously considered unverifiable, testing whether they appropriately revise their judgments when the evidentiary situation changes.
The benchmark uses independently constructed ground truth and evaluates correct and missed findings, false positives, incorrect conclusions, unsupported inference, uncertainty handling, and revision of previous judgments.
The complete benchmark case is publicly available, including the model-facing evidence, additional evidence, structured data, model responses, answer keys, evaluation materials, Cardinal patch files, and Python comparison tooling.
Repository: https://github.com/apparent-symmetry/EvidenceBench
I believe it may be a useful addition to the list because it evaluates a somewhat different aspect of multimodal/agent reasoning: not simply whether an agent can produce a technically plausible answer, but whether its conclusions are actually supported by the evidence available to it.
Thank you for your time.
I’d like to suggest adding EvidenceBench, a benchmark I created, to your list of agent benchmarks.
EvidenceBench is a multimodal AI benchmark designed to evaluate whether models can distinguish what is actually supported by visual evidence from what is merely technically plausible.
The initial benchmark, CardinalBenchmark-001 (cb-001), uses a deliberately complex Cardinal (VCV Rack) modular-synthesis patch as a controlled evaluation environment. The goal is not to test whether a model knows how a particular synthesizer module works, but whether it can maintain accurate information about modules, cables, spatial relationships, and technical connections when the visual environment becomes difficult to interpret.
CB-001 uses a three-turn evaluation:
The benchmark uses independently constructed ground truth and evaluates correct and missed findings, false positives, incorrect conclusions, unsupported inference, uncertainty handling, and revision of previous judgments.
The complete benchmark case is publicly available, including the model-facing evidence, additional evidence, structured data, model responses, answer keys, evaluation materials, Cardinal patch files, and Python comparison tooling.
Repository: https://github.com/apparent-symmetry/EvidenceBench
I believe it may be a useful addition to the list because it evaluates a somewhat different aspect of multimodal/agent reasoning: not simply whether an agent can produce a technically plausible answer, but whether its conclusions are actually supported by the evidence available to it.
Thank you for your time.