Skip to content

Committee-size sweep: dilution and proportional pressure (MedMCQA) #276

Description

@duckyquang

Replicate the Committee-size sweep: dilution and proportional pressure (MedMCQA) experiment on MedMCQA, mirroring the MedQA template so results are apples-to-apples across datasets.

What it measures

Single fixed wrong seed (dilution) vs fixed wrong fraction (proportional pressure) across committee sizes.

What to do

Run the same experiment on MedMCQA through the existing pipeline and the Gemini API. Replication, not new features: do not change the experiment code, only point it at a MedMCQA manifest (build one with the medmcqa adapter). Write results under experiments/medmcqa/results/ in the same JSON/JSONL convention as experiments/medqa/results/ (not CSV, so the analysis tooling reads them).

Acceptance

  • Results committed under experiments/medmcqa/ in the MedQA template's format.
  • The same headline statistic the template reports, on MedMCQA.
  • Verified against a real API run (no synthetic data, no smoke-test-only claims).
  • PR opened that closes this issue.

Part of the MedMCQA standard-battery replication (mirrors the MedQA/NIH template).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions