From 6c11f7133a46fc2a562b23f5cbb5ad987b6a6340 Mon Sep 17 00:00:00 2001 From: James Kane Date: Thu, 30 Jul 2026 12:10:27 -0500 Subject: [PATCH] Archaic design doc: record the shipped state and where it diverged from the plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc's header still read "No code yet" for a feature that shipped in v0.1.0-alpha.14, M3 was headed "in progress" after M3.3 wired it end to end, and §9's open questions were headed "Still open before Phase 1" with all three long since closed. Corrected, and §8 now carries per-phase status. The staleness was the easy half. Three things shipped DIFFERENTLY from §10 and none was recorded where the plan states the rule, so a reader following the document would have been misled about what the code does: - M3 mandates a feature gate over Tier B covering the read AND publish paths. No such constant exists. Extent shipped ungated; only lineage attribution is gated. Defensible -- the fabrication risk the gate existed to contain was entirely in attribution -- but the plan still read as if the gate were there. - §7's "~1.5-2% Neanderthal" is not the target the caller was calibrated against. Three denominators are in play (whole genome, callable genome, hmmix's per-haplotype union) and they are not interchangeable: the shipped claim is 91.5 Mb against hmmix's measured EUR mean of 90.9 Mb, which is 1.01x. Quoting §7's figure next to the shipped 5.04% compares two different measurements, so §7 now says so in place. - M3 requires a test asserting "none reliably detected" for the Denisovan floor. It was never written. Shipped behaviour is stronger (Denisovan is never surfaced at all), but nothing currently FAILS if a future change starts emitting denisovan_copies as a headline. That is an open gap, not a deviation that resolved itself, and it is recorded as one. Adds a "Deviations from the plan" section carrying those three with their justification and their residual risk -- extent is fitted on ONE individual, so its error is uncharacterised for anyone with unusual ancestry or coverage. Plus an open-follow-ups list: the missing test, multi-individual validation, the Skov-2020 attribution upgrade, and the asset-staging decision. That last one is a live defect rather than a doc bug. M1 said to check Asset 1's size against ON_DEMAND_PREFIXES "before deciding" and no decision was ever made, so PATTERNS lists no archaic_* while release mode bundles all five (105.3 MB) -- the two staging modes disagree, and alpha.14's installers are ~60 MB larger than alpha.13's. Recorded here; fixed separately. Documentation only; no code changes. Co-Authored-By: Claude Opus 5 (1M context) --- documents/design/ArchaicAncestry_Design.md | 96 ++++++++++++++++++++-- 1 file changed, 88 insertions(+), 8 deletions(-) diff --git a/documents/design/ArchaicAncestry_Design.md b/documents/design/ArchaicAncestry_Design.md index 7c41488..d8e6906 100644 --- a/documents/design/ArchaicAncestry_Design.md +++ b/documents/design/ArchaicAncestry_Design.md @@ -1,8 +1,10 @@ # Archaic Ancestry Report (Neanderthal / Denisovan) — Design -**Status:** Design + **implementation plan (§10)**. No code yet. Drafted 2026-07-23; plan added -2026-07-26 on branch `feat/archaic-ancestry`. Two of the three §9 open questions are now settled; -the third is a gating checkpoint inside M1. +**Status:** **SHIPPED** in `v0.1.0-alpha.14` (2026-07-30). Tier A (#34) and Tier B (#35) are both in +`main` and in users' hands; §8 Phases 1 and 2 are complete, Phase 3 (M4) is not started and remains +optional. Drafted 2026-07-23; plan added 2026-07-26 on branch `feat/archaic-ancestry`; all three §9 +questions resolved. **Two things shipped differently from the plan below — see *Deviations from the +plan* at the end of §10 before trusting §7's expected percentage or M3's feature-gate rule.** **Goal:** Reconstruct a 23andMe-style Neanderthal report — and go beyond it with a Denisovan estimate and a true whole-genome introgression map — from public archaic reference genomes and recent methods, using the app's existing ancestry/panel/HMM machinery. @@ -222,6 +224,15 @@ Verified population baselines: a typical **European carries ~1.5–2% Neandertha in the Icelandic data Denisovan was 3.3% *of* the ~2% archaic total, i.e. a European's Denisovan signal sits **at the noise floor**. Denisovan ancestry concentrates in Oceanians / East & South Asians. +> **As shipped, the ~1.5–2 % figure above is NOT the number the caller targets.** It is the commonly +> quoted whole-genome fraction from the literature, and Tier B was ultimately calibrated against +> hmmix's *measured* 1000G European distribution instead — mean **90.9 Mb** (p10 84.6, p90 97.2, +> n=633), which is ~2.9 % of a 3.1 Gb genome, not 1.5–2 %. Our shipped result is 91.5 Mb = 5.04 % of +> **callable** (1,815 Mb), a third denominator again. Three different denominators are in play and +> they are not comparable: whole genome, callable genome, and hmmix's per-haplotype-unioned total. +> The validation claim is **1.01× hmmix's mean**, and nothing in this section supersedes that. See +> *Deviations from the plan* (§10). + Design consequence: **never emit a fabricated small Denisovan number for a European.** This is the same failure mode that got ancient ancestry disabled ([[ancient-ancestry-broken]] — fabricated numbers from centroids that sat on top of each other). Follow the `estimate_qpadm_ancestry` precedent: report @@ -234,6 +245,9 @@ against his real 23andMe Neanderthal count if available. ## 8. Phasing +*Status as of v0.1.0-alpha.14: **Phase 1 DONE**, **Phase 2 DONE** (with per-segment Nea/Den +attribution built but gated off), **Phase 3 NOT STARTED**.* + - **Phase 1 (MVP):** Asset 1 + Asset 4, Tier A `count_archaic_markers`, domain/store/UI card. Delivers the full 23andMe-equivalent for **both chip and WGS**, reusing the ancestry-panel machinery almost verbatim. Low risk. Validate the marker count against James's actual 23andMe report. @@ -261,11 +275,12 @@ The **hmmix Zenodo set (CC BY 4.0)** is retained as a citable cross-validation r carries into Phase 1: finalize the exact site-selection thresholds (AFR-frequency cutoff, min non-African frequency) by calibrating against that cross-validation set. -### Still open before Phase 1 +### Resolved during Phase 1 (this heading formerly read "Still open before Phase 1") -*(Reviewed 2026-07-26 when the implementation plan below was written; two of the three are now settled.)* +*(Reviewed 2026-07-26 when the implementation plan below was written; two of the three were settled +then. **All three are now closed** — Q1 was measured at M1 checkpoint B, below.)* -1. **Chip overlap — MEASUREMENT SCHEDULED (M1 checkpoint B).** How many informative sites actually +1. **Chip overlap — MEASURED, see *Checkpoint B* in §10.** How many informative sites actually overlap 23andMe/AncestryDNA v5 chip content after lift to CHM13? Unanswerable until candidate sites exist, so it becomes a gating checkpoint *inside* M1 rather than a blocker before it. It sets the realistic chip call rate and the honest ceiling on the count. @@ -392,11 +407,22 @@ disabled. Tier B has more moving parts than Tier A and the same blast radius, so discipline — a constant gate that also covers the read *and* publish paths, flipped only once the validation targets below pass. +> **NOT IMPLEMENTED AS WRITTEN.** No such constant exists. Tier B's *extent* shipped ungated — the +> "Archaic segments" card is always present — and only the **lineage attribution** is gated +> (`attribute_lineage: false`). See *Deviations from the plan* below for why, and for what the +> resulting exposure actually is. + **Validation targets:** ~1.5–2 % Neanderthal and **no** Denisovan for the European ground-truth sample (§7); the overall Nea/Den/unknown split should be in the neighbourhood of Skov 2020's 84.5 / 3.3 / 12.2 on Icelanders. The Denisovan confidence floor must be exercised by a test that asserts "none reliably detected" rather than a small number. +> **SUPERSEDED, and one part unmet.** The extent target became hmmix's measured EUR distribution +> rather than §7's 1.5–2 % (see the note in §7 and *Extent — CALIBRATED* below). The Nea/Den/unknown +> split target is moot: attribution is off, so everything reports unattributed. **The required +> "none reliably detected" test was never written** — Denisovan is instead never surfaced at all, +> which is a stronger guarantee but not the check specified here. + ### M1 — as built (2026-07-28) The pipeline ran end to end. Inputs: 202.8 GB of EVA all-sites VCFs (four genomes × 22 autosomes) @@ -716,7 +742,10 @@ rank. Exactness would need the cohort's per-sample genotypes at runtime (~188 MB shape `HaplotypeReference` already uses); that is the upgrade path if the approximation ever proves insufficient. -### M3 (Tier B) — in progress: assets + HMM built, callability mask REQUIRED +### M3 (Tier B) — DONE: assets + HMM built, callability mask REQUIRED + +*(This section was written as a running log while M3 was in progress; it reads chronologically and +ends at the shipped state. M3.1–M3.3 are all complete and shipped in v0.1.0-alpha.14.)* Built and unit-tested: the two Tier B assets (`ArchaicOutgroup`, `ArchaicClassify` — delta-varint position streams, 1.07 bytes/site measured, so the ~67 M-position outgroup track lands near 72 MB @@ -876,14 +905,65 @@ Three decisions, each about refusing to do the wrong thing silently: support (see the attribution section). Segment width is floored at 1.5 px because a 31 kb median tract is sub-pixel on a whole-chromosome track. -### M4 — Phase 3 (optional) +### M4 — Phase 3 (optional) — NOT STARTED Trait associations (each needs a curated GWAS-backed table — curatorial work, not engineering), export + PDS records, fine-pop percentile re-keying, and a possible DAIseg-style joint Nea/Den upgrade if that method matures past preprint. +Partial credit only: the archaic count reaches the **HTML brief export** (`export.rs:506`), so the +exported artifact agrees with the UI. There is no archaic **PDS/federated record**, no trait table, +and the percentile is still keyed to the 1kGP super-population as decided in §9 Q3. + ### Sequencing note M1 is the long pole and everything else depends on Asset 1, but M2 is where the user-visible feature lands. M3 is separable — if effort runs short, **M1 + M2 is a complete, honest, shippable report** (it is exactly what 23andMe ships), and Tier B can follow later without rework. + +--- + +## Deviations from the plan (recorded 2026-07-30, post-ship audit) + +Three things shipped differently from §10 as written. Each is defensible; none was written down at +the point where the plan states the rule, which is why this section exists. A plan that silently +disagrees with the code is worse than no plan, because the next reader trusts it. + +**1. Tier B shipped without the feature gate M3 mandates.** The plan called for a constant covering +the read *and* publish paths, flipped only after the validation targets passed. What exists is +narrower: `attribute_lineage: false` suppresses the Neanderthal-vs-Denisovan split, while the extent +number (Mb, % of callable) and the chromosome browser are always available. + +*Why this is defensible:* the gate's purpose was to stop a fabricated number reaching users, and the +fabrication risk was concentrated entirely in attribution — which is gated. Extent was calibrated +against an external callset and lands at 1.01× hmmix's European mean, so it passed the bar the gate +was protecting. *What the exposure actually is:* extent is fitted on **one individual**, so a user +with unusual ancestry or coverage gets a number whose error is uncharacterised. That is a real and +unmeasured risk, not a resolved one. + +**2. §7's ~1.5–2 % is not the shipped target.** Three denominators are in circulation — whole genome, +callable genome, and hmmix's per-haplotype-unioned total — and they are not interchangeable. The +operative claim is 91.5 Mb against hmmix's measured EUR mean of 90.9 Mb. §7 now carries a note +saying so. Anyone quoting "1.5–2 %" from this document alongside the shipped 5.04 % is comparing two +different measurements. + +**3. The Denisovan "none reliably detected" test was never written.** The shipped behaviour is +stronger than the spec — Denisovan is never surfaced as a finding at all, and `archaic.denisovanNote` +in the UI explains why — but the specific regression test M3 requires does not exist. Nothing +currently fails if a future change starts emitting `denisovan_copies` as a headline. **This is a +genuine open gap, not a deviation that resolved itself.** + +### Open follow-ups + +- **Write the Denisovan-floor regression test** (item 3 above). +- **Validate extent across several individuals**, not one. Needs WGS for people whose hmmix result is + known. Until then the 1.01× agreement is a calibration check, not a validation. +- **Lineage attribution** needs the Skov-2020 approach — match a segment's own haplotype against each + archaic genome relative to a background expectation, rather than against pre-classified site + categories. Threshold tuning will not fix it; see the attribution section above. +- **Asset staging policy was never decided.** M1 said to check Asset 1's size against + `ON_DEMAND_PREFIXES` in `packaging/stage-assets.sh` "before deciding"; that decision was never + made. As a result `PATTERNS` lists no `archaic_*`, so dev-mode staging omits them, while + release-mode fetches everything the manifest names except `ancestry_haps_` and bundles all five + (105.3 MB). The two modes disagree — exactly what the comment above `PATTERNS` warns against — and + alpha.14's installers are ~60 MB larger than alpha.13's as a result. Tracked separately.