Skip to content

Real-run results report and reproducibility bundle (per-arm) #105

Description

@sebasmos

Problem

We have a transcript store (#89), a run-manifest writer (#23), and an HTML report generator (#88), but no single convention for what a completed real run of one arm leaves on disk so that a result is reproducible and reviewable months later. As the real runs start (MedQA now, imaging next) we need every run to drop a self-contained bundle.

What to build

Define and implement a per-run output bundle written by the run entrypoint (#):

results/<arm>-<dataset>-<date>/
  config.json         # fully resolved config (models, cues, seeds, limits, decoding)
  versions.json       # package version + pinned deps + git SHA (issue #21)
  manifest.json       # run manifest (issue #23)
  results.json        # the runner's structured output
  summary.md          # human-readable: pilot bite, susceptibility table, overlap, cascade onset/contagion
  transcripts/        # committed transcripts for replay (issue #24/#25)
  • summary.md is the artifact a collaborator reads first: go/no-go pilot signal, per-model x per-cue susceptibility, same vs cross-lineage overlap with p-value, cascade onset count and mean contagion.
  • The bundle must be enough to re-run analysis offline from transcripts/ with no model calls (deterministic replay).

Acceptance

  • Running any stage produces the bundle above.
  • A benchmaxxing report <bundle-dir> renders the HTML report (HTML results report generator #88) purely from the bundle.
  • A test builds a bundle with a mock backend and asserts every file is present and parses, and that replay from transcripts/ reproduces the analysis.

Do not commit real-run bundles that contain dataset-derived text into the repo; keep them under DATASET_ROOT. Only fixtures/mock bundles live in the repo.

Metadata

Metadata

Assignees

Labels

difficulty: intermediateTouches one subsystem; some context neededdocumentationImprovements or additions to documentationpriority: mediumValuable, not on the critical path

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions