You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a transcript store (#89), a run-manifest writer (#23), and an HTML report generator (#88), but no single convention for what a completed real run of one arm leaves on disk so that a result is reproducible and reviewable months later. As the real runs start (MedQA now, imaging next) we need every run to drop a self-contained bundle.
What to build
Define and implement a per-run output bundle written by the run entrypoint (#):
summary.md is the artifact a collaborator reads first: go/no-go pilot signal, per-model x per-cue susceptibility, same vs cross-lineage overlap with p-value, cascade onset count and mean contagion.
The bundle must be enough to re-run analysis offline from transcripts/ with no model calls (deterministic replay).
A test builds a bundle with a mock backend and asserts every file is present and parses, and that replay from transcripts/ reproduces the analysis.
Do not commit real-run bundles that contain dataset-derived text into the repo; keep them under DATASET_ROOT. Only fixtures/mock bundles live in the repo.
Problem
We have a transcript store (#89), a run-manifest writer (#23), and an HTML report generator (#88), but no single convention for what a completed real run of one arm leaves on disk so that a result is reproducible and reviewable months later. As the real runs start (MedQA now, imaging next) we need every run to drop a self-contained bundle.
What to build
Define and implement a per-run output bundle written by the
runentrypoint (#):summary.mdis the artifact a collaborator reads first: go/no-go pilot signal, per-model x per-cue susceptibility, same vs cross-lineage overlap with p-value, cascade onset count and mean contagion.transcripts/with no model calls (deterministic replay).Acceptance
benchmaxxing report <bundle-dir>renders the HTML report (HTML results report generator #88) purely from the bundle.transcripts/reproduces the analysis.Do not commit real-run bundles that contain dataset-derived text into the repo; keep them under
DATASET_ROOT. Only fixtures/mock bundles live in the repo.