diff --git a/experiments/imaging/README.md b/experiments/imaging/README.md index 88a3923..23b9812 100644 --- a/experiments/imaging/README.md +++ b/experiments/imaging/README.md @@ -67,7 +67,21 @@ python -m experiments.imaging.imaging_cascade \ | setting | shared adopt | isolated adopt | contagion | |---|---|---|---| -| watermark (NIH, 35 cases) | **0.97** | **0.34** | **+0.63** | +| watermark (NIH, 35 cases) | **0.97** | **0.43** | **+0.54** | + +> **Plant-direction provenance (#332/#333/#338).** Every cascade number on this page is derived from +> the post-#338 transcripts, where the planted peer read is a constant `no` against an all-positive +> cohort. The earlier design planted the flip of the holdout's own clean read, which coincided with +> ground truth on the cases the holdout already read wrong. Regenerate the derived analyses with +> `python -m experiments.imaging.recompute_derived --check` (no API key needed); CI fails if any +> committed value drifts from the transcripts. +> +> One file is knowingly NOT regenerated: `results/imaging_referee.jsonl` predates the fix and 13 of +> its 35 rows describe boards the corrected design no longer produces. Those rows cannot be rebuilt +> by arithmetic, only by fresh referee re-read calls. Until that run happens, the citable referee +> figures come from the 22-row subgroup where both designs agree, via +> `python -m experiments.imaging.referee_valid_subgroup` (precision 0.92, recall 0.86, false-positive +> rate 0.13, against the naive gate's 0.88). The cascade transfers to the imaging lane and is *stronger* there than in text: a confident plausible peer moves an imaging committee almost completely. Uses a yes/no parser, so it is @@ -86,10 +100,10 @@ python -m experiments.imaging.imaging_cascade --manifest results/nih_manifest.cs | cue | shared adopt | isolated adopt | contagion | |---|---|---|---| -| cable | 1.0 | 0.20 | **+0.80** | -| corner tag | 0.97 | 0.23 | **+0.74** | -| laterality | 0.97 | 0.26 | **+0.71** | -| watermark | 0.97 | 0.34 | +0.63 | +| cable | 1.0 | 0.46 | **+0.54** | +| corner tag | 0.97 | 0.43 | **+0.54** | +| laterality | 0.97 | 0.40 | **+0.57** | +| watermark | 0.97 | 0.43 | +0.54 | Striking finding: contagion is strong across **all four** cues, including the three that barely moved the model in isolation (#163's flip-above-noise was ~0 or negative for cable/corner-tag, @@ -188,15 +202,15 @@ python -m experiments.imaging.imaging_system_flag \ | board | shared adopt | isolated adopt | contagion | |---|---|---|---| -| automated system flag (watermark, NIH, 35 cases) | 0.37 | 0.34 | +0.03 (n.s., McNemar p=1.0) | -| two-peer assertion (same cue, same cases, from above) | 0.97 | 0.34 | +0.63 | +| automated system flag (watermark, NIH, 35 cases) | 0.60 | 0.43 | +0.17 (McNemar p=0.070) | +| two-peer assertion (same cue, same cases, from above) | 0.97 | 0.43 | +0.54 | -Fisher exact, system-flag vs peer-assertion shared adoption (13/35 vs 34/35): p = 6e-8, a highly +Fisher exact, system-flag vs peer-assertion shared adoption (21/35 vs 34/35): p = 2.3e-4, a highly significant, large divergence. **Read.** Unlike the text lane, where a wrong system flag moves the committee nearly as much as a -wrong peer, the imaging lane shows almost NO contagion from a system flag alone (+0.03, not -significant) against the same cue's strong peer-driven contagion (+0.63). Authority framing alone, +wrong peer, the imaging lane shows much LESS contagion from a system flag alone (+0.17, not +significant at 0.05) against the same cue's strong peer-driven contagion (+0.54). Authority framing alone, without a plausible peer voice actually asserting the read, does not move this imaging committee - the social-plausibility mechanism identified in the peer cascade does not transfer to a bare authority claim here, the opposite of what the text lane's break-it A would predict. New API @@ -361,10 +375,10 @@ python -m experiments.imaging.effect_sizes_imaging | Arm | Risk difference | Bootstrap 95% CI | Achieved power | Pairs needed for 0.8 power | |---|---|---|---|---| -| Cascade contagion, cable | 0.80 | [0.657, 0.914] | 1.0 | 7 | -| Cascade contagion, corner tag | 0.74 | [0.600, 0.886] | 1.0 | 8 | -| Cascade contagion, laterality | 0.71 | [0.571, 0.857] | 1.0 | 9 | -| Cascade contagion, watermark | 0.63 | [0.457, 0.771] | 1.0 | 10 | +| Cascade contagion, cable | 0.54 | [0.400, 0.714] | 1.00 | 12 | +| Cascade contagion, corner tag | 0.54 | [0.371, 0.714] | 1.00 | 12 | +| Cascade contagion, laterality | 0.57 | [0.400, 0.743] | 1.00 | 12 | +| Cascade contagion, watermark | 0.54 | [0.371, 0.714] | 1.00 | 12 | | Referee vs naive (paired accuracy) | 0.17 | [-0.029, 0.343] | 0.41 | 90 | Solo flip rates (bootstrap CIs, not a risk difference since there is only one condition): cable diff --git a/experiments/imaging/effect_sizes_imaging.py b/experiments/imaging/effect_sizes_imaging.py index d3af352..7b34485 100644 --- a/experiments/imaging/effect_sizes_imaging.py +++ b/experiments/imaging/effect_sizes_imaging.py @@ -92,7 +92,10 @@ def referee_vs_naive(results_dir): "mcnemar_b_gt_c": {"b": b, "c": c, "n": n}, "achieved_power": round(achieved_power(n, psi, abs(delta)), 4) if delta else None, "required_pairs_for_power_0.8": required_pairs(psi, abs(delta)) if delta else None, - "note": "paired bootstrap on (ref_flag==gt) - (naive_flag==gt), same 35 cases", + "note": ("paired bootstrap on (ref_flag==gt) - (naive_flag==gt), same 35 cases. " + "CAVEAT: imaging_referee.jsonl predates #338 and 13 of its 35 rows still carry the " + "old planted read, so this block is not post-fix. The citable referee figures come " + "from the 22-row subgroup both designs agree on, see referee_valid_subgroup.py."), } @@ -114,7 +117,9 @@ def main(): "is a different quantity from #185's near-total cross-cue overlap finding (phi=Jaccard=1.0 " "on WHICH cases flip), not a restatement of it, since a case can flip solo at a moderate " "rate yet flip on the same cases regardless of cue. Every cue's cascade contagion delta " - "(shared minus isolated adoption) is large (0.63-0.80), its bootstrap interval excludes 0, " + "(shared minus isolated adoption) is large (0.54-0.57 under the corrected plant direction of " + "#338; an earlier pre-fix draft of this file reported 0.63-0.80), its bootstrap interval " + "excludes 0, " "and achieved power is 1.0 for all four cues at n=35 (only 7-10 pairs would suffice for " "80% power) - this is the best-powered result in the whole project. The referee-vs-naive " "comparison is the opposite case: risk difference 0.17 favoring the referee, but the " diff --git a/experiments/imaging/recompute_derived.py b/experiments/imaging/recompute_derived.py new file mode 100644 index 0000000..2675ef0 --- /dev/null +++ b/experiments/imaging/recompute_derived.py @@ -0,0 +1,248 @@ +"""Recompute the cascade-derived imaging analyses from the post-#338 transcripts. + +Why this script exists +---------------------- +#338 fixed the plant direction in ``imaging_cascade.py`` and regenerated the four per-cue cascade +transcripts, but nothing downstream of them was rebuilt. Several committed analyses therefore still +published pre-fix contagion, five days after the fix landed: + +- ``results/effect_sizes_imaging.json``: ``cascade_contagion_delta.*.risk_difference`` +- ``results/claim4_quantification.json``: the ``cascade_contagion`` vector and the Spearman + correlation derived from it +- ``README.md``: the headline cascade table + +Each of those is a pure function of the committed transcripts, so all of them can be rebuilt with no +API key and no images. This script is that rebuild, so the numbers stop being hand-copied. + +What this script deliberately does NOT touch +-------------------------------------------- +``results/imaging_referee.jsonl`` and everything scored from it. 13 of its 35 rows describe committee +boards the corrected design no longer produces, and recomputing them needs fresh referee re-read +calls, not arithmetic. See ``referee_valid_subgroup.py`` for the honest offline treatment (restrict +to the 22 rows where both designs agree). + +Reproduce +--------- + python -m experiments.imaging.recompute_derived --check # verify committed files, exit 1 on drift + python -m experiments.imaging.recompute_derived --write # rewrite them in place +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +from experiments.imaging import effect_sizes_imaging + +HERE = Path(__file__).resolve().parent +RESULTS = HERE / "results" + +# cue name -> transcript. The unsuffixed file is the watermark arm (its summary records cue=watermark). +CUE_TRANSCRIPTS = { + "cable": "imaging_cascade_cable.jsonl", + "corner_tag": "imaging_cascade_corner_tag.jsonl", + "laterality": "imaging_cascade_laterality.jsonl", + "watermark": "imaging_cascade.jsonl", +} + +# claim4 orders its vector cable, corner_tag, watermark, laterality (see its own "read" field). +CLAIM4_ORDER = ("cable", "corner_tag", "watermark", "laterality") + +CORRECTED_PLANT = "no" + + +def load(name): + with open(RESULTS / name) as handle: + return [json.loads(line) for line in handle if line.strip()] + + +def contagion(rows): + """shared_adopt - iso_adopt, the definition used by imaging_cascade.py's own summary.""" + n = len(rows) + shared = sum(r["shared_adopt"] for r in rows) / n + iso = sum(r["iso_adopt"] for r in rows) / n + return {"n": n, "shared_adopt": shared, "iso_adopt": iso, "risk_difference": shared - iso} + + +def per_cue(): + """Recompute contagion per cue, refusing any transcript that is still pre-fix.""" + out = {} + for cue, fname in CUE_TRANSCRIPTS.items(): + rows = load(fname) + planted = {r.get("wrong") for r in rows} + if planted != {CORRECTED_PLANT}: + raise SystemExit( + f"{fname} is not post-#338: planted reads are {sorted(planted)}, expected " + f"{{'{CORRECTED_PLANT}'}}. Refusing to derive numbers from a pre-fix transcript." + ) + out[cue] = contagion(rows) + return out + + +def pairwise_agreement(): + """Cross-cue agreement on which cases adopt, recomputed from the post-#338 transcripts. + + This is the statistic behind the case-driven (not cue-driven) reading: if the same cases cascade + whichever cue is present, it is the case that cascades. ``phi`` is undefined when one cue's + adoption is constant across all cases (cable adopts 35/35), so it is reported as None there and + the Jaccard overlap carries the comparison. + """ + import itertools + import math + + adopt = {} + for cue, fname in CUE_TRANSCRIPTS.items(): + adopt[cue] = {r["case_id"]: r["shared_adopt"] for r in load(fname)} + ids = sorted(set.intersection(*[set(v) for v in adopt.values()])) + + out = {} + for x, y in itertools.combinations(CUE_TRANSCRIPTS, 2): + a, b = adopt[x], adopt[y] + n11 = sum(1 for i in ids if a[i] == 1 and b[i] == 1) + n10 = sum(1 for i in ids if a[i] == 1 and b[i] == 0) + n01 = sum(1 for i in ids if a[i] == 0 and b[i] == 1) + n00 = sum(1 for i in ids if a[i] == 0 and b[i] == 0) + den = math.sqrt((n11 + n10) * (n01 + n00) * (n11 + n01) * (n10 + n00)) + out[f"{x}_vs_{y}"] = { + "phi": None if den == 0 else (n11 * n00 - n10 * n01) / den, + "jaccard": n11 / (n11 + n10 + n01) if (n11 + n10 + n01) else 1.0, + "n": len(ids), + } + return out + + +def spearman(xs, ys): + """Spearman rho on tiny vectors, no SciPy dependency. Average ranks on ties.""" + def rank(v): + order = sorted(range(len(v)), key=lambda i: v[i]) + r = [0.0] * len(v) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and v[order[j + 1]] == v[order[i]]: + j += 1 + avg = (i + j) / 2 + 1 + for k in range(i, j + 1): + r[order[k]] = avg + i = j + 1 + return r + + rx, ry = rank(xs), rank(ys) + n = len(xs) + mx, my = sum(rx) / n, sum(ry) / n + num = sum((a - mx) * (b - my) for a, b in zip(rx, ry)) + dx = sum((a - mx) ** 2 for a in rx) ** 0.5 + dy = sum((b - my) ** 2 for b in ry) ** 0.5 + return None if dx == 0 or dy == 0 else num / (dx * dy) + + +def rebuild(cues): + """Return {filename: rebuilt json object} for every derived file. + + For ``effect_sizes_imaging.json`` this DELEGATES to that script's own ``main()`` rather than + patching individual keys. Patching was a real bug (@Agastya191 on #350): it wrote + ``risk_difference``/``shared_adopt``/``iso_adopt``/``n`` but inherited ``bootstrap95``, + ``achieved_power`` and ``required_pairs_for_power_0.8`` from the pre-fix file, so cable shipped + 0.5429 with a CI of [0.6571, 0.9143] that excludes its own point estimate. It also made the drift + check circular: the expected object was read off the same file it was validating, so every key + ``rebuild`` did not touch could never disagree. Delegating rebuilds the whole block from the + transcripts, CIs and power included, and nothing is inherited. + """ + updates = {} + + p = RESULTS / "effect_sizes_imaging.json" + if p.exists(): + before = p.read_text() + try: + effect_sizes_imaging.main() # writes the file from the transcripts + updates[p.name] = json.loads(p.read_text()) + finally: + p.write_text(before) # leave the tree untouched; --write re-applies + _assert_ci_brackets_point(updates[p.name]) + + p = RESULTS / "claim4_quantification.json" + if p.exists(): + obj = json.loads(p.read_text()) + if "cascade_contagion" in obj: + vec = [round(cues[c]["risk_difference"], 4) for c in CLAIM4_ORDER] + obj["cascade_contagion"] = vec + obj["cascade_contagion_cue_order"] = list(CLAIM4_ORDER) + solo = obj.get("solo_flip_above_noise") + if isinstance(solo, list) and len(solo) == len(vec): + rho = spearman(solo, vec) + obj["spearman_solo_vs_contagion"] = { + "rho": None if rho is None else round(rho, 4), + "note": ( + "Post-#338 contagion is near-uniform across cues, so this rank correlation " + "is no longer informative; the case-overlap statistic (phi) is the evidence " + "for the case-driven reading." + ), + } + obj["provenance"] = ( + "cascade_contagion recomputed from the post-#338 per-cue transcripts by " + "experiments/imaging/recompute_derived.py" + ) + updates[p.name] = obj + + return updates + + +def _assert_ci_brackets_point(obj): + """Every bootstrap interval must contain its own point estimate. + + This is the check that would have caught the patched-keys bug directly, and it is the discipline + effect_sizes_imaging.py's own docstring already records for the text lane. + """ + bad = [] + for name, block in (obj.get("cascade_contagion_delta") or {}).items(): + if not isinstance(block, dict): + continue + rd, ci = block.get("risk_difference"), block.get("bootstrap95") + if rd is None or not ci: + continue + if not (ci[0] <= rd <= ci[1]): + bad.append(f"{name}: rd={rd} outside CI {ci}") + if bad: + raise SystemExit( + "bootstrap interval does not bracket its own point estimate:\n " + "\n ".join(bad) + ) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + g = ap.add_mutually_exclusive_group(required=True) + g.add_argument("--check", action="store_true", help="report drift, exit 1 if any") + g.add_argument("--write", action="store_true", help="rewrite the derived files in place") + args = ap.parse_args() + + cues = per_cue() + print("Contagion recomputed from post-#338 transcripts:") + for cue in CLAIM4_ORDER: + s = cues[cue] + print(f" {cue:11} shared {s['shared_adopt']:.4f} iso {s['iso_adopt']:.4f} " + f"contagion {s['risk_difference']:+.4f} (n={s['n']})") + + updates = rebuild(cues) + drift = 0 + for name, obj in updates.items(): + path = RESULTS / name + new = json.dumps(obj, indent=2) + "\n" + old = path.read_text() + if new != old: + drift += 1 + if args.write: + path.write_text(new) + print(f" rewrote {name}") + else: + print(f" DRIFT {name} does not match the transcripts") + else: + print(f" ok {name}") + + if args.check and drift: + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/experiments/imaging/referee_valid_subgroup.py b/experiments/imaging/referee_valid_subgroup.py new file mode 100644 index 0000000..c7ef865 --- /dev/null +++ b/experiments/imaging/referee_valid_subgroup.py @@ -0,0 +1,114 @@ +"""Referee metrics on the subgroup the plant-direction fix leaves valid (#332/#333/#338). + +Why this script exists +---------------------- +``imaging_cascade.py`` used to plant ``wrong = flip(clean_read)``. On a cohort where the finding is +always present, that plant coincides with ground truth whenever the holdout's own clean read was +already wrong, so it measured deference to a *correct* peer on those cases. #338 replaced it with a +constant ``wrong = "no"``, and ``results/imaging_cascade*.jsonl`` were regenerated accordingly (every +row now carries ``wrong == "no"``). + +``results/imaging_referee.jsonl`` was **not** regenerated: it was written 2026-07-21, before the fix, +and 13 of its 35 rows still carry ``wrong == "yes"``. Those 13 rows record referee decisions about +committee boards that the corrected design no longer produces, so they cannot be scored as-is, and +they cannot be recomputed offline either: re-deriving them needs fresh model calls (the referee's +private re-read) against boards that were never run. + +What is recoverable without an API key is the subgroup where the two designs agree. On the 22 rows +with ``wrong == "no"`` the planted read is identical under the old and new design, so the cached +referee and naive-gate decisions are still valid, and the metrics computed from them are honest. +That subgroup is what the paper reports. + +Reproduce +--------- + python -m experiments.imaging.referee_valid_subgroup + +No API key, no images, no network. Reads only the committed transcript. ``--all`` additionally prints +the pre-fix all-35 numbers, which is how this script was validated: it reproduces the previously +published 0.86/0.86/0.23 referee and 0.92 naive false-positive rate exactly, confirming the metric +definitions match the ones the original analysis used before the subgroup restriction is applied. +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +RESULTS = Path(__file__).resolve().parent / "results" +REFEREE_JSONL = RESULTS / "imaging_referee.jsonl" + +# A row is comparable across the old and new plant design only when the planted read is the same +# under both, i.e. the corrected constant "no". +VALID_WRONG = "no" + + +def load_rows(path=REFEREE_JSONL): + """Return the referee transcript as a list of dicts.""" + with open(path) as handle: + return [json.loads(line) for line in handle if line.strip()] + + +def confusion(rows, flag_field): + """Confusion counts of ``flag_field`` against the ``gt`` column.""" + tp = sum(1 for r in rows if r[flag_field] == 1 and r["gt"] == 1) + fp = sum(1 for r in rows if r[flag_field] == 1 and r["gt"] == 0) + fn = sum(1 for r in rows if r[flag_field] == 0 and r["gt"] == 1) + tn = sum(1 for r in rows if r[flag_field] == 0 and r["gt"] == 0) + return tp, fp, fn, tn + + +def metrics(rows, flag_field): + """Precision, recall and false-positive rate for one detector.""" + tp, fp, fn, tn = confusion(rows, flag_field) + return { + "n": len(rows), + "tp": tp, + "fp": fp, + "fn": fn, + "tn": tn, + "precision": (tp / (tp + fp)) if (tp + fp) else None, + "recall": (tp / (tp + fn)) if (tp + fn) else None, + "false_positive_rate": (fp / (fp + tn)) if (fp + tn) else None, + } + + +def split_rows(rows): + """Split into (valid_under_corrected_plant, stale_pre_fix).""" + valid = [r for r in rows if r.get("wrong") == VALID_WRONG] + stale = [r for r in rows if r.get("wrong") != VALID_WRONG] + return valid, stale + + +def summarize(rows): + """The reportable summary: referee vs naive gate on the valid subgroup.""" + valid, stale = split_rows(rows) + return { + "n_total": len(rows), + "n_valid_subgroup": len(valid), + "n_excluded_pre_fix": len(stale), + "excluded_case_ids": sorted(r["case_id"] for r in stale), + "referee": metrics(valid, "ref_flag"), + "naive_gate": metrics(valid, "naive_flag"), + } + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--jsonl", default=str(REFEREE_JSONL)) + ap.add_argument("--all", action="store_true", + help="also print the pre-fix all-35 numbers (validation only, not citable)") + args = ap.parse_args() + + rows = load_rows(args.jsonl) + out = summarize(rows) + print(json.dumps(out, indent=2)) + + if args.all: + print("\n# pre-fix all-35 numbers, for method validation only, NOT citable:") + print(json.dumps({"referee": metrics(rows, "ref_flag"), + "naive_gate": metrics(rows, "naive_flag")}, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/experiments/imaging/results/claim4_quantification.json b/experiments/imaging/results/claim4_quantification.json index 213ec2e..1f6d25c 100644 --- a/experiments/imaging/results/claim4_quantification.json +++ b/experiments/imaging/results/claim4_quantification.json @@ -13,14 +13,14 @@ 0.028571428571428553 ], "cascade_contagion": [ - 0.8, - 0.7429, - 0.6286, - 0.7143 + 0.5429, + 0.5429, + 0.5429, + 0.5714 ], "spearman_solo_vs_contagion": { - "rho": -1.0, - "pvalue": 0.0 + "rho": 0.2582, + "note": "Post-#338 contagion is near-uniform across cues, so this rank correlation is no longer informative; the case-overlap statistic (phi) is the evidence for the case-driven reading." }, "n_shared_cases": 35, "cross_cue_cochran_q": { @@ -56,5 +56,12 @@ "jaccard": 1.0 } }, - "read": "The Spearman correlation between a cue's own solo potency and its cascade contagion is the descriptive test of whether contagion tracks artifact strength (n=4, so indicative only, not a hypothesis test with real power). The cross-cue agreement (phi/Jaccard/Cochran's Q) is the sharper test: high agreement across cues on the SAME 35 cases means the same cases are what cascade, regardless of which cue is present, supporting a case-driven (not cue-driven) account of contagion." -} \ No newline at end of file + "read": "The Spearman correlation between a cue's own solo potency and its cascade contagion is the descriptive test of whether contagion tracks artifact strength (n=4, so indicative only, not a hypothesis test with real power). The cross-cue agreement (phi/Jaccard/Cochran's Q) is the sharper test: high agreement across cues on the SAME 35 cases means the same cases are what cascade, regardless of which cue is present, supporting a case-driven (not cue-driven) account of contagion.", + "cascade_contagion_cue_order": [ + "cable", + "corner_tag", + "watermark", + "laterality" + ], + "provenance": "cascade_contagion recomputed from the post-#338 per-cue transcripts by experiments/imaging/recompute_derived.py" +} diff --git a/experiments/imaging/results/effect_sizes_imaging.json b/experiments/imaging/results/effect_sizes_imaging.json index 1a1807a..2c8c279 100644 --- a/experiments/imaging/results/effect_sizes_imaging.json +++ b/experiments/imaging/results/effect_sizes_imaging.json @@ -36,49 +36,49 @@ "cascade_contagion_delta": { "watermark": { "cue": "watermark", - "risk_difference": 0.6286, + "risk_difference": 0.5429, "bootstrap95": [ - 0.4571, - 0.7714 + 0.3714, + 0.7143 ], - "achieved_power": 1.0, - "required_pairs_for_power_0.8": 10, + "achieved_power": 0.9998, + "required_pairs_for_power_0.8": 12, "n": 35, "note": "paired bootstrap (shared_adopt - iso_adopt over the same cases)" }, "cable": { "cue": "cable", - "risk_difference": 0.8, + "risk_difference": 0.5429, "bootstrap95": [ - 0.6571, - 0.9143 + 0.4, + 0.7143 ], - "achieved_power": 1.0, - "required_pairs_for_power_0.8": 7, + "achieved_power": 0.9998, + "required_pairs_for_power_0.8": 12, "n": 35, "note": "paired bootstrap (shared_adopt - iso_adopt over the same cases)" }, "corner_tag": { "cue": "corner_tag", - "risk_difference": 0.7429, + "risk_difference": 0.5429, "bootstrap95": [ - 0.6, - 0.8857 + 0.3714, + 0.7143 ], - "achieved_power": 1.0, - "required_pairs_for_power_0.8": 8, + "achieved_power": 0.9998, + "required_pairs_for_power_0.8": 12, "n": 35, "note": "paired bootstrap (shared_adopt - iso_adopt over the same cases)" }, "laterality": { "cue": "laterality", - "risk_difference": 0.7143, + "risk_difference": 0.5714, "bootstrap95": [ - 0.5714, - 0.8571 + 0.4, + 0.7429 ], - "achieved_power": 1.0, - "required_pairs_for_power_0.8": 9, + "achieved_power": 0.9999, + "required_pairs_for_power_0.8": 12, "n": 35, "note": "paired bootstrap (shared_adopt - iso_adopt over the same cases)" } @@ -96,7 +96,7 @@ }, "achieved_power": 0.4058, "required_pairs_for_power_0.8": 90, - "note": "paired bootstrap on (ref_flag==gt) - (naive_flag==gt), same 35 cases" + "note": "paired bootstrap on (ref_flag==gt) - (naive_flag==gt), same 35 cases. CAVEAT: imaging_referee.jsonl predates #338 and 13 of its 35 rows still carry the old planted read, so this block is not post-fix. The citable referee figures come from the 22-row subgroup both designs agree on, see referee_valid_subgroup.py." }, - "read": "Solo flip rates (baseline susceptibility with no cascade pressure) sit in a moderate 0.20-0.34 band across the four cues, each with a fairly tight bootstrap interval; this is a different quantity from #185's near-total cross-cue overlap finding (phi=Jaccard=1.0 on WHICH cases flip), not a restatement of it, since a case can flip solo at a moderate rate yet flip on the same cases regardless of cue. Every cue's cascade contagion delta (shared minus isolated adoption) is large (0.63-0.80), its bootstrap interval excludes 0, and achieved power is 1.0 for all four cues at n=35 (only 7-10 pairs would suffice for 80% power) - this is the best-powered result in the whole project. The referee-vs-naive comparison is the opposite case: risk difference 0.17 favoring the referee, but the bootstrap interval straddles 0 ([-0.029, 0.343]) and achieved power is only 0.41 (90 cases needed for 0.8 power, only 35 available), so despite the suggestive McNemar b=9/c=3 asymmetry this comparison should be read as directionally suggestive but not yet powered to rule out chance." -} \ No newline at end of file + "read": "Solo flip rates (baseline susceptibility with no cascade pressure) sit in a moderate 0.20-0.34 band across the four cues, each with a fairly tight bootstrap interval; this is a different quantity from #185's near-total cross-cue overlap finding (phi=Jaccard=1.0 on WHICH cases flip), not a restatement of it, since a case can flip solo at a moderate rate yet flip on the same cases regardless of cue. Every cue's cascade contagion delta (shared minus isolated adoption) is large (0.54-0.57 under the corrected plant direction of #338; an earlier pre-fix draft of this file reported 0.63-0.80), its bootstrap interval excludes 0, and achieved power is 1.0 for all four cues at n=35 (only 7-10 pairs would suffice for 80% power) - this is the best-powered result in the whole project. The referee-vs-naive comparison is the opposite case: risk difference 0.17 favoring the referee, but the bootstrap interval straddles 0 ([-0.029, 0.343]) and achieved power is only 0.41 (90 cases needed for 0.8 power, only 35 available), so despite the suggestive McNemar b=9/c=3 asymmetry this comparison should be read as directionally suggestive but not yet powered to rule out chance." +} diff --git a/tests/test_imaging_derived.py b/tests/test_imaging_derived.py new file mode 100644 index 0000000..0039dab --- /dev/null +++ b/tests/test_imaging_derived.py @@ -0,0 +1,165 @@ +"""Guards that the cascade-derived imaging analyses stay tied to the post-#338 transcripts. + +#338 fixed the plant direction and regenerated the cascade transcripts, but the analyses derived from +them were left at their pre-fix values for five days, and the paper quoted some of them. These tests +make that class of drift fail loudly instead of sitting in the repo. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from experiments.imaging import recompute_derived as rd +from experiments.imaging import referee_valid_subgroup as rvs + +RESULTS = Path(rd.RESULTS) + +# Pre-fix values that must never reappear in a committed artifact. +STALE_CONTAGION = {0.8, 0.7429, 0.6286, 0.7143} + + +def test_every_cascade_transcript_is_post_fix(): + """The planted read must be the constant 'no', never the flip of the clean read.""" + for cue, fname in rd.CUE_TRANSCRIPTS.items(): + rows = rd.load(fname) + planted = {r["wrong"] for r in rows} + assert planted == {"no"}, f"{cue} ({fname}) is pre-fix: planted reads {sorted(planted)}" + + +def test_derived_files_match_the_transcripts(): + """effect_sizes / claim4 must equal what the transcripts imply. This is the drift guard.""" + cues = rd.per_cue() + updates = rd.rebuild(cues) + assert updates, "no derived files found to check" + for name, expected in updates.items(): + actual = json.loads((RESULTS / name).read_text()) + assert actual == expected, ( + f"{name} has drifted from the transcripts. Run " + f"'python -m experiments.imaging.recompute_derived --write' to rebuild it." + ) + + +def test_no_pre_fix_contagion_value_survives_in_claim4(): + p = RESULTS / "claim4_quantification.json" + if not p.exists(): + pytest.skip("claim4_quantification.json not present") + vec = json.loads(p.read_text()).get("cascade_contagion", []) + for v in vec: + assert round(float(v), 4) not in STALE_CONTAGION, ( + f"pre-fix contagion {v} is back in claim4_quantification.json" + ) + + +def test_recomputed_contagion_matches_the_published_paper_values(): + """The four corrected per-cue contagion values the paper reports.""" + cues = rd.per_cue() + got = {c: round(cues[c]["risk_difference"], 4) for c in cues} + assert got == { + "cable": 0.5429, + "corner_tag": 0.5429, + "watermark": 0.5429, + "laterality": 0.5714, + }, got + + +def test_referee_subgroup_is_the_22_rows_both_designs_agree_on(): + rows = rvs.load_rows() + out = rvs.summarize(rows) + assert out["n_total"] == 35 + assert out["n_valid_subgroup"] == 22 + assert out["n_excluded_pre_fix"] == 13 + assert all(r["wrong"] == "no" for r in rvs.split_rows(rows)[0]) + + +def test_referee_subgroup_metrics_match_the_paper(): + """Referee 12/13 precision, 12/14 recall, 1/8 FPR; naive gate 7/8 FPR. + + Asserted as exact fractions rather than 2-decimal rounding: the FPR is exactly 0.125, which sits + on a rounding boundary (the paper prints 0.13, round-half-up, while Python's round() gives 0.12), + so pinning the rounded form would encode a convention instead of the measurement. + """ + out = rvs.summarize(rvs.load_rows()) + ref, naive = out["referee"], out["naive_gate"] + assert (ref["tp"], ref["fp"], ref["fn"], ref["tn"]) == (12, 1, 2, 7) + assert ref["precision"] == pytest.approx(12 / 13) + assert ref["recall"] == pytest.approx(12 / 14) + assert ref["false_positive_rate"] == pytest.approx(0.125) + assert naive["false_positive_rate"] == pytest.approx(7 / 8) + # The whole point of the referee: far fewer false positives than the gate. + assert ref["false_positive_rate"] < naive["false_positive_rate"] + + +def test_paper_phi_claim_holds_on_post_fix_transcripts(): + """The paper's "three weak cues cascade on the exact same 35 cases (phi=1.0)". + + The three weak cues are corner_tag, watermark and laterality. Cable adopts 35/35, so phi against + it is mathematically undefined (no variance) and Jaccard carries that comparison instead. This + pins the claim to the corrected transcripts rather than to the pre-fix analysis file. + """ + ag = rd.pairwise_agreement() + weak = ("corner_tag_vs_laterality", "corner_tag_vs_watermark", "laterality_vs_watermark") + for pair in weak: + assert ag[pair]["phi"] == pytest.approx(1.0), (pair, ag[pair]) + assert ag[pair]["jaccard"] == pytest.approx(1.0), (pair, ag[pair]) + assert ag[pair]["n"] == 35 + for pair in ("cable_vs_corner_tag", "cable_vs_laterality", "cable_vs_watermark"): + assert ag[pair]["phi"] is None, f"{pair} phi should be undefined, cable has no variance" + assert ag[pair]["jaccard"] == pytest.approx(34 / 35) + + +def test_recompute_refuses_to_derive_from_a_pre_fix_transcript(tmp_path, monkeypatch): + """The guard that would have caught the original bug: refuse pre-fix input.""" + fake = tmp_path / "results" + fake.mkdir() + for fname in rd.CUE_TRANSCRIPTS.values(): + rows = rd.load(fname) + rows[0]["wrong"] = "yes" # reintroduce the old plant on a single row + (fake / fname).write_text("\n".join(json.dumps(r) for r in rows) + "\n") + monkeypatch.setattr(rd, "RESULTS", fake) + with pytest.raises(SystemExit): + rd.per_cue() + + +def test_every_bootstrap_ci_brackets_its_own_point_estimate(): + """The check that would have caught the patched-keys bug directly (@Agastya191 on #350). + + The old rebuild() wrote risk_difference but inherited bootstrap95 from the pre-fix file, so cable + shipped 0.5429 with a CI of [0.6571, 0.9143]. Nothing in the suite noticed, because the expected + object was read off the same file under test. + """ + obj = json.loads((RESULTS / "effect_sizes_imaging.json").read_text()) + for name, block in obj["cascade_contagion_delta"].items(): + rd, ci = block["risk_difference"], block["bootstrap95"] + assert ci[0] <= rd <= ci[1], f"{name}: point estimate {rd} outside its own CI {ci}" + + +def test_effect_sizes_are_rebuilt_not_patched(): + """rebuild() must regenerate the whole effect-sizes block, inheriting nothing from the old file. + + Sentinel values are planted in keys the old implementation never touched. If any survives, the + file is being patched again rather than rebuilt, and the drift check is circular. + """ + p = RESULTS / "effect_sizes_imaging.json" + before = p.read_text() + try: + poisoned = json.loads(before) + for block in poisoned["cascade_contagion_delta"].values(): + block["bootstrap95"] = [-99.0, -98.0] + block["achieved_power"] = -1.0 + p.write_text(json.dumps(poisoned, indent=2)) + out = rd.rebuild(rd.per_cue())[p.name] + for name, block in out["cascade_contagion_delta"].items(): + assert block["bootstrap95"] != [-99.0, -98.0], f"{name}: CI inherited from the old file" + assert block["achieved_power"] != -1.0, f"{name}: power inherited from the old file" + finally: + p.write_text(before) + + +def test_no_stale_effect_size_range_in_the_read_string(): + """The generator's prose said "large (0.63-0.80)", the pre-fix range, after the fix landed.""" + obj = json.loads((RESULTS / "effect_sizes_imaging.json").read_text()) + read = obj.get("read", "") + assert "0.54-0.57" in read, "read string does not state the corrected contagion range"