fix(imaging): rebuild the cascade-derived analyses from the post-#338 transcripts

experiments/imaging/README.md

@@ -67,7 +67,21 @@ python -m experiments.imaging.imaging_cascade \
 
 | setting | shared adopt | isolated adopt | contagion |
 |---|---|---|---|
-| watermark (NIH, 35 cases) | **0.97** | **0.34** | **+0.63** |
+| watermark (NIH, 35 cases) | **0.97** | **0.43** | **+0.54** |
+
+> **Plant-direction provenance (#332/#333/#338).** Every cascade number on this page is derived from
+> the post-#338 transcripts, where the planted peer read is a constant `no` against an all-positive
+> cohort. The earlier design planted the flip of the holdout's own clean read, which coincided with
+> ground truth on the cases the holdout already read wrong. Regenerate the derived analyses with
+> `python -m experiments.imaging.recompute_derived --check` (no API key needed); CI fails if any
+> committed value drifts from the transcripts.
+>
+> One file is knowingly NOT regenerated: `results/imaging_referee.jsonl` predates the fix and 13 of
+> its 35 rows describe boards the corrected design no longer produces. Those rows cannot be rebuilt
+> by arithmetic, only by fresh referee re-read calls. Until that run happens, the citable referee
+> figures come from the 22-row subgroup where both designs agree, via
+> `python -m experiments.imaging.referee_valid_subgroup` (precision 0.92, recall 0.86, false-positive
+> rate 0.13, against the naive gate's 0.88).
 
 The cascade transfers to the imaging lane and is *stronger* there than in text: a confident
 plausible peer moves an imaging committee almost completely. Uses a yes/no parser, so it is
@@ -86,10 +100,10 @@ python -m experiments.imaging.imaging_cascade --manifest results/nih_manifest.cs
 
 | cue | shared adopt | isolated adopt | contagion |
 |---|---|---|---|
-| cable | 1.0 | 0.20 | **+0.80** |
-| corner tag | 0.97 | 0.23 | **+0.74** |
-| laterality | 0.97 | 0.26 | **+0.71** |
-| watermark | 0.97 | 0.34 | +0.63 |
+| cable | 1.0 | 0.46 | **+0.54** |
+| corner tag | 0.97 | 0.43 | **+0.54** |
+| laterality | 0.97 | 0.40 | **+0.57** |
+| watermark | 0.97 | 0.43 | +0.54 |
 
 Striking finding: contagion is strong across **all four** cues, including the three that barely
 moved the model in isolation (#163's flip-above-noise was ~0 or negative for cable/corner-tag,
@@ -188,15 +202,15 @@ python -m experiments.imaging.imaging_system_flag \
 
 | board | shared adopt | isolated adopt | contagion |
 |---|---|---|---|
-| automated system flag (watermark, NIH, 35 cases) | 0.37 | 0.34 | +0.03 (n.s., McNemar p=1.0) |
-| two-peer assertion (same cue, same cases, from above) | 0.97 | 0.34 | +0.63 |
+| automated system flag (watermark, NIH, 35 cases) | 0.60 | 0.43 | +0.17 (McNemar p=0.070) |
+| two-peer assertion (same cue, same cases, from above) | 0.97 | 0.43 | +0.54 |
 
-Fisher exact, system-flag vs peer-assertion shared adoption (13/35 vs 34/35): p = 6e-8, a highly
+Fisher exact, system-flag vs peer-assertion shared adoption (21/35 vs 34/35): p = 2.3e-4, a highly
 significant, large divergence.
 
 **Read.** Unlike the text lane, where a wrong system flag moves the committee nearly as much as a
-wrong peer, the imaging lane shows almost NO contagion from a system flag alone (+0.03, not
-significant) against the same cue's strong peer-driven contagion (+0.63). Authority framing alone,
+wrong peer, the imaging lane shows much LESS contagion from a system flag alone (+0.17, not
+significant at 0.05) against the same cue's strong peer-driven contagion (+0.54). Authority framing alone,
 without a plausible peer voice actually asserting the read, does not move this imaging committee -
 the social-plausibility mechanism identified in the peer cascade does not transfer to a bare
 authority claim here, the opposite of what the text lane's break-it A would predict. New API
@@ -361,10 +375,10 @@ python -m experiments.imaging.effect_sizes_imaging
 
 | Arm | Risk difference | Bootstrap 95% CI | Achieved power | Pairs needed for 0.8 power |
 |---|---|---|---|---|
-| Cascade contagion, cable | 0.80 | [0.657, 0.914] | 1.0 | 7 |
-| Cascade contagion, corner tag | 0.74 | [0.600, 0.886] | 1.0 | 8 |
-| Cascade contagion, laterality | 0.71 | [0.571, 0.857] | 1.0 | 9 |
-| Cascade contagion, watermark | 0.63 | [0.457, 0.771] | 1.0 | 10 |
+| Cascade contagion, cable | 0.54 | [0.400, 0.714] | 1.00 | 12 |
+| Cascade contagion, corner tag | 0.54 | [0.371, 0.714] | 1.00 | 12 |
+| Cascade contagion, laterality | 0.57 | [0.400, 0.743] | 1.00 | 12 |
+| Cascade contagion, watermark | 0.54 | [0.371, 0.714] | 1.00 | 12 |
 | Referee vs naive (paired accuracy) | 0.17 | [-0.029, 0.343] | 0.41 | 90 |
 
 Solo flip rates (bootstrap CIs, not a risk difference since there is only one condition): cable

experiments/imaging/effect_sizes_imaging.py

@@ -92,7 +92,10 @@ def referee_vs_naive(results_dir):
         "mcnemar_b_gt_c": {"b": b, "c": c, "n": n},
         "achieved_power": round(achieved_power(n, psi, abs(delta)), 4) if delta else None,
         "required_pairs_for_power_0.8": required_pairs(psi, abs(delta)) if delta else None,
-        "note": "paired bootstrap on (ref_flag==gt) - (naive_flag==gt), same 35 cases",
+        "note": ("paired bootstrap on (ref_flag==gt) - (naive_flag==gt), same 35 cases. "
+                 "CAVEAT: imaging_referee.jsonl predates #338 and 13 of its 35 rows still carry the "
+                 "old planted read, so this block is not post-fix. The citable referee figures come "
+                 "from the 22-row subgroup both designs agree on, see referee_valid_subgroup.py."),
     }
 
 
@@ -114,7 +117,9 @@ def main():
             "is a different quantity from #185's near-total cross-cue overlap finding (phi=Jaccard=1.0 "
             "on WHICH cases flip), not a restatement of it, since a case can flip solo at a moderate "
             "rate yet flip on the same cases regardless of cue. Every cue's cascade contagion delta "
-            "(shared minus isolated adoption) is large (0.63-0.80), its bootstrap interval excludes 0, "
+            "(shared minus isolated adoption) is large (0.54-0.57 under the corrected plant direction of "
+            "#338; an earlier pre-fix draft of this file reported 0.63-0.80), its bootstrap interval "
+            "excludes 0, "
             "and achieved power is 1.0 for all four cues at n=35 (only 7-10 pairs would suffice for "
             "80% power) - this is the best-powered result in the whole project. The referee-vs-naive "
             "comparison is the opposite case: risk difference 0.17 favoring the referee, but the "

experiments/imaging/recompute_derived.py

@@ -0,0 +1,248 @@
+"""Recompute the cascade-derived imaging analyses from the post-#338 transcripts.
+
+Why this script exists
+----------------------
+#338 fixed the plant direction in ``imaging_cascade.py`` and regenerated the four per-cue cascade
+transcripts, but nothing downstream of them was rebuilt. Several committed analyses therefore still
+published pre-fix contagion, five days after the fix landed:
+
+- ``results/effect_sizes_imaging.json``: ``cascade_contagion_delta.*.risk_difference``
+- ``results/claim4_quantification.json``: the ``cascade_contagion`` vector and the Spearman
+  correlation derived from it
+- ``README.md``: the headline cascade table
+
+Each of those is a pure function of the committed transcripts, so all of them can be rebuilt with no
+API key and no images. This script is that rebuild, so the numbers stop being hand-copied.
+
+What this script deliberately does NOT touch
+--------------------------------------------
+``results/imaging_referee.jsonl`` and everything scored from it. 13 of its 35 rows describe committee
+boards the corrected design no longer produces, and recomputing them needs fresh referee re-read
+calls, not arithmetic. See ``referee_valid_subgroup.py`` for the honest offline treatment (restrict
+to the 22 rows where both designs agree).
+
+Reproduce
+---------
+    python -m experiments.imaging.recompute_derived --check   # verify committed files, exit 1 on drift
+    python -m experiments.imaging.recompute_derived --write   # rewrite them in place
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from experiments.imaging import effect_sizes_imaging
+
+HERE = Path(__file__).resolve().parent
+RESULTS = HERE / "results"
+
+# cue name -> transcript. The unsuffixed file is the watermark arm (its summary records cue=watermark).
+CUE_TRANSCRIPTS = {
+    "cable": "imaging_cascade_cable.jsonl",
+    "corner_tag": "imaging_cascade_corner_tag.jsonl",
+    "laterality": "imaging_cascade_laterality.jsonl",
+    "watermark": "imaging_cascade.jsonl",
+}
+
+# claim4 orders its vector cable, corner_tag, watermark, laterality (see its own "read" field).
+CLAIM4_ORDER = ("cable", "corner_tag", "watermark", "laterality")
+
+CORRECTED_PLANT = "no"
+
+
+def load(name):
+    with open(RESULTS / name) as handle:
+        return [json.loads(line) for line in handle if line.strip()]
+
+
+def contagion(rows):
+    """shared_adopt - iso_adopt, the definition used by imaging_cascade.py's own summary."""
+    n = len(rows)
+    shared = sum(r["shared_adopt"] for r in rows) / n
+    iso = sum(r["iso_adopt"] for r in rows) / n
+    return {"n": n, "shared_adopt": shared, "iso_adopt": iso, "risk_difference": shared - iso}
+
+
+def per_cue():
+    """Recompute contagion per cue, refusing any transcript that is still pre-fix."""
+    out = {}
+    for cue, fname in CUE_TRANSCRIPTS.items():
+        rows = load(fname)
+        planted = {r.get("wrong") for r in rows}
+        if planted != {CORRECTED_PLANT}:
+            raise SystemExit(
+                f"{fname} is not post-#338: planted reads are {sorted(planted)}, expected "
+                f"{{'{CORRECTED_PLANT}'}}. Refusing to derive numbers from a pre-fix transcript."
+            )
+        out[cue] = contagion(rows)
+    return out
+
+
+def pairwise_agreement():
+    """Cross-cue agreement on which cases adopt, recomputed from the post-#338 transcripts.
+
+    This is the statistic behind the case-driven (not cue-driven) reading: if the same cases cascade
+    whichever cue is present, it is the case that cascades. ``phi`` is undefined when one cue's
+    adoption is constant across all cases (cable adopts 35/35), so it is reported as None there and
+    the Jaccard overlap carries the comparison.
+    """
+    import itertools
+    import math
+
+    adopt = {}
+    for cue, fname in CUE_TRANSCRIPTS.items():
+        adopt[cue] = {r["case_id"]: r["shared_adopt"] for r in load(fname)}
+    ids = sorted(set.intersection(*[set(v) for v in adopt.values()]))
+
+    out = {}
+    for x, y in itertools.combinations(CUE_TRANSCRIPTS, 2):
+        a, b = adopt[x], adopt[y]
+        n11 = sum(1 for i in ids if a[i] == 1 and b[i] == 1)
+        n10 = sum(1 for i in ids if a[i] == 1 and b[i] == 0)
+        n01 = sum(1 for i in ids if a[i] == 0 and b[i] == 1)
+        n00 = sum(1 for i in ids if a[i] == 0 and b[i] == 0)
+        den = math.sqrt((n11 + n10) * (n01 + n00) * (n11 + n01) * (n10 + n00))
+        out[f"{x}_vs_{y}"] = {
+            "phi": None if den == 0 else (n11 * n00 - n10 * n01) / den,
+            "jaccard": n11 / (n11 + n10 + n01) if (n11 + n10 + n01) else 1.0,
+            "n": len(ids),
+        }
+    return out
+
+
+def spearman(xs, ys):
+    """Spearman rho on tiny vectors, no SciPy dependency. Average ranks on ties."""
+    def rank(v):
+        order = sorted(range(len(v)), key=lambda i: v[i])
+        r = [0.0] * len(v)
+        i = 0
+        while i < len(order):
+            j = i
+            while j + 1 < len(order) and v[order[j + 1]] == v[order[i]]:
+                j += 1
+            avg = (i + j) / 2 + 1
+            for k in range(i, j + 1):
+                r[order[k]] = avg
+            i = j + 1
+        return r
+
+    rx, ry = rank(xs), rank(ys)
+    n = len(xs)
+    mx, my = sum(rx) / n, sum(ry) / n
+    num = sum((a - mx) * (b - my) for a, b in zip(rx, ry))
+    dx = sum((a - mx) ** 2 for a in rx) ** 0.5
+    dy = sum((b - my) ** 2 for b in ry) ** 0.5
+    return None if dx == 0 or dy == 0 else num / (dx * dy)
+
+
+def rebuild(cues):
+    """Return {filename: rebuilt json object} for every derived file.
+
+    For ``effect_sizes_imaging.json`` this DELEGATES to that script's own ``main()`` rather than
+    patching individual keys. Patching was a real bug (@Agastya191 on #350): it wrote
+    ``risk_difference``/``shared_adopt``/``iso_adopt``/``n`` but inherited ``bootstrap95``,
+    ``achieved_power`` and ``required_pairs_for_power_0.8`` from the pre-fix file, so cable shipped
+    0.5429 with a CI of [0.6571, 0.9143] that excludes its own point estimate. It also made the drift
+    check circular: the expected object was read off the same file it was validating, so every key
+    ``rebuild`` did not touch could never disagree. Delegating rebuilds the whole block from the
+    transcripts, CIs and power included, and nothing is inherited.
+    """
+    updates = {}
+
+    p = RESULTS / "effect_sizes_imaging.json"
+    if p.exists():
+        before = p.read_text()
+        try:
+            effect_sizes_imaging.main()          # writes the file from the transcripts
+            updates[p.name] = json.loads(p.read_text())
+        finally:
+            p.write_text(before)                 # leave the tree untouched; --write re-applies
+        _assert_ci_brackets_point(updates[p.name])
+
+    p = RESULTS / "claim4_quantification.json"
+    if p.exists():
+        obj = json.loads(p.read_text())
+        if "cascade_contagion" in obj:
+            vec = [round(cues[c]["risk_difference"], 4) for c in CLAIM4_ORDER]
+            obj["cascade_contagion"] = vec
+            obj["cascade_contagion_cue_order"] = list(CLAIM4_ORDER)
+            solo = obj.get("solo_flip_above_noise")
+            if isinstance(solo, list) and len(solo) == len(vec):
+                rho = spearman(solo, vec)
+                obj["spearman_solo_vs_contagion"] = {
+                    "rho": None if rho is None else round(rho, 4),
+                    "note": (
+                        "Post-#338 contagion is near-uniform across cues, so this rank correlation "
+                        "is no longer informative; the case-overlap statistic (phi) is the evidence "
+                        "for the case-driven reading."
+                    ),
+                }
+            obj["provenance"] = (
+                "cascade_contagion recomputed from the post-#338 per-cue transcripts by "
+                "experiments/imaging/recompute_derived.py"
+            )
+            updates[p.name] = obj
+
+    return updates
+
+
+def _assert_ci_brackets_point(obj):
+    """Every bootstrap interval must contain its own point estimate.
+
+    This is the check that would have caught the patched-keys bug directly, and it is the discipline
+    effect_sizes_imaging.py's own docstring already records for the text lane.
+    """
+    bad = []
+    for name, block in (obj.get("cascade_contagion_delta") or {}).items():
+        if not isinstance(block, dict):
+            continue
+        rd, ci = block.get("risk_difference"), block.get("bootstrap95")
+        if rd is None or not ci:
+            continue
+        if not (ci[0] <= rd <= ci[1]):
+            bad.append(f"{name}: rd={rd} outside CI {ci}")
+    if bad:
+        raise SystemExit(
+            "bootstrap interval does not bracket its own point estimate:\n  " + "\n  ".join(bad)
+        )
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
+    g = ap.add_mutually_exclusive_group(required=True)
+    g.add_argument("--check", action="store_true", help="report drift, exit 1 if any")
+    g.add_argument("--write", action="store_true", help="rewrite the derived files in place")
+    args = ap.parse_args()
+
+    cues = per_cue()
+    print("Contagion recomputed from post-#338 transcripts:")
+    for cue in CLAIM4_ORDER:
+        s = cues[cue]
+        print(f"  {cue:11} shared {s['shared_adopt']:.4f}  iso {s['iso_adopt']:.4f}  "
+              f"contagion {s['risk_difference']:+.4f}  (n={s['n']})")
+
+    updates = rebuild(cues)
+    drift = 0
+    for name, obj in updates.items():
+        path = RESULTS / name
+        new = json.dumps(obj, indent=2) + "\n"
+        old = path.read_text()
+        if new != old:
+            drift += 1
+            if args.write:
+                path.write_text(new)
+                print(f"  rewrote {name}")
+            else:
+                print(f"  DRIFT {name} does not match the transcripts")
+        else:
+            print(f"  ok {name}")
+
+    if args.check and drift:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()