MDStudyBench

A benchmark asking whether an agent can use molecular dynamics to answer a scientific question. There is no curator-authored workflow to imitate and no preferred structure: the agent plans the study, and the evaluation fixes only the estimand, the replayed observable, the decision rule, and the validity controls.

The active dataset is MDStudyBench-v0.4 with one experimental pilot task, S01: does raising hydrostatic pressure from 0.1 MPa to 200 MPa increase, decrease, or leave unchanged the equilibrium hydration of the engineered internal cavity in folded T4 lysozyme L99A?

Its preparation-focused sibling, MDPrepBench, lives at matsunagalab/MDPrepBench. Both were extracted from matsunagalab/mdclaw.

The grounded-correct contract

The primary result is one bit:

grounded_correct = valid_execution AND claim_supported AND truth_agreement

The gates do not compensate, and no LLM judge contributes. The flow has two agent handoffs:

1. The agent prepares systems, creates pending MDClaw production nodes, writes submission/confirmatory_plan.json, and exits. It never runs confirmatory MD itself.
2. The benchmark runner freezes the plan, validates the paired systems, and executes the nodes through a certified adapter against a frozen MDClaw source snapshot.
3. The agent is re-invoked (a fresh process in the same workspace; up to two continuations when the plan needs repair), reads only the runner-certified episode artifacts, writes submission/claim.json, and exits.

The evaluator replays the cavity-water observable and folded-state control from the runner-certified trajectories; agent self-reports never enter the score.

Installation

git clone https://github.com/matsunagalab/MDStudyBench
cd MDStudyBench
pip install -e .
mdstudybench --list

Two runtime requirements, deliberately separate:

• Scoring needs an interpreter that can import openmm, mdtraj, and numpy (conda-forge is easiest). When the local interpreter lacks them and MDCLAW_SIF points at a Singularity image, the harness delegates scoring into that container.
• Confirmatory execution additionally needs MDClaw: the mdclaw package must be importable by the harness interpreter (the runner freezes and attests the MDClaw source it executes), and an MDClaw runtime — conda env, SIF, or docker image — must be available for the adapter. Scoring an existing submission needs neither.

Running an agent

mdstudybench run_benchmark_agent \
  --dataset-dir benchmarks/mdstudybench \
  --output-dir benchmark_runs \
  --run-id <run_id> \
  --agent-name <agent> \
  --max-walltime-minutes-per-task 0 \
  --judge-mode deterministic

Any agent plugs in through --agent-command; the built-in profiles cover claude-code, codex, and pi. Warning: those profiles launch the CLIs with approval/sandbox bypass flags, because a benchmark run must proceed without interactive confirmation. Run them only in an environment you are willing to let an agent modify.

The task's declared time_limit_minutes (24 h for S01) is authoritative for study tasks; the operator cap can shorten but never extend it.

Openness

Everything is public, including the task contract and the held-out experimental truth under tasks/S01_*/truth/. The benchmark measures whether an agent can resolve the question with valid MD, not whether it can guess a hidden rubric — but agents being evaluated must only ever see the exported public package (prompt.md, submission_contract.json, submission_checklist.md), which the harness prepares in each solver workspace. Solver isolation is the operator's responsibility.

Repository layout note

This is a checkout-based application: the dataset and operator tools live in the repository tree, and a bare pip install mdstudybench wheel carries only the harness package. Clone the repository to use the benchmark.

Development

ruff check mdstudybench/ tests/ benchmarks/
python -m pytest tests/ -q          # scoring tests need openmm+mdtraj
python benchmarks/mdstudybench/scripts/generate_tasks.py --check

Task contracts are generated from benchmarks/mdstudybench/task_specs/; edit the specs, not tasks/*/task.json. docs/memo.md is the running record of benchmark work.

Status

S01 is a non-primary pilot while the runner-owned OpenMM adapter and blind feasibility are calibrated. Known limits are stated in the dataset notes: the pilot attests production relative to the frozen base System; base-system construction and the complete dependency runtime remain unattested diagnostics.

License

The code is MIT-licensed. Experimental reference data under tasks/*/truth/ derives from published literature cited in the task contracts.