Skip to content

[W07] Restate reported numbers from the canonical run #68

Description

@duckyquang

Task metadata

Field Value
Task ID W07
Category Writing
Priority Medium
Complexity Low
Est. days 2
Blocked by E07, E08, E09, E10

Features / method

Replace transcribed estimates in the docs with the canonical result set.

Description

Once E07-E10 land, every number in the repo needs to be restated from the canonical run. Right now the README results table carries n=245 from an earlier cohort build, while the manifest in the repo has a 248-row test split, so the docs describe a cohort that no longer matches the code.

Also worth a pass for correctness while in there: the README and TECHNICAL.md both still describe the fairness and calibration analyses as part of the pipeline, and until E08/E09 produce artifacts those read as stronger than what exists.

Steps:

  1. Update the README results table with canonical numbers plus CIs, and correct n.
  2. Update the provenance note — drop the "transcribed from the E02 evaluation" caveat once it no longer applies, and state the device and command that produced the run.
  3. Update TECHNICAL.md 8 with in-cohort solo baselines alongside the published ones, so the comparison is not misread as like-for-like.
  4. Check off the resolved items in TECHNICAL.md 11 (pooling, pairing window, masking strategy) and delete any that E06/E07/D07 closed.
  5. Add fairness and calibration results to the README, or state plainly that they are pending.
  6. Confirm no synthetic harness-validation output from failure/demo.py has leaked into any results table. Those numbers look real and are not.

Acceptance criteria:

  • Every reported number traceable to a run that was actually executed, with device and command recorded.
  • No transcribed or estimated values presented as measured.
  • Cohort n consistent between docs and the manifest in the repo.
  • Synthetic demo output clearly labelled wherever it appears.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions