From 57bda736bf937385daa3c264222a2947cc01a500 Mon Sep 17 00:00:00 2001 From: James Kane Date: Thu, 30 Jul 2026 12:24:43 -0500 Subject: [PATCH] Decide the archaic asset staging split; both modes were disagreeing silently `archaic_*` appeared in NEITHER `PATTERNS` nor `ON_DEMAND_PREFIXES`. Those two lists are the same decision written once per source mode -- local-dir mode bundles what PATTERNS names, release mode bundles everything the manifest names EXCEPT ON_DEMAND_PREFIXES -- so an asset missing from both is bundled in CI and absent from dev builds. Nobody chose that; the archaic design's M1 said to check Asset 1's size against the on-demand policy "before deciding" and no decision was ever recorded. Cost of the omission, measured on the release it shipped in: all five archaic assets (105.3 MB) went into v0.1.0-alpha.14, whose installers came out ~60 MB larger than alpha.13's -- .dmg 136 -> 199 MB, .exe 105 -> 163 MB. The split, on the same principle as `ancestry_haps_`: bundle what users can reach, download what almost nobody does. - Tier A (`archaic_markers_`, `archaic_marker_dist_`, 33 MB) BUNDLED. The marker count runs on chip data as well as WGS, so every user can reach it and it has to work with no network. This is also what M1 assumed. - Tier B (`archaic_outgroup_af_`, `archaic_classify_`, `archaic_callable_`, 72 MB) ON DEMAND. Segment calling needs whole-genome data AND an hours-long de-novo calling pass to have already completed, so the reachable population is very small for two thirds of the archaic payload. Verified rather than assumed: - Simulated BOTH modes against the real manifest: each now stages the identical 13 files / 312.2 MB, divergence empty in both directions. Before this change release mode staged 384.2 MB and local mode 312.2 MB. - Ran the script: 14 files staged (13 + manifest), Tier A present, Tier B and ancestry_haps absent. - The download path for all three Tier B assets already exists and is on the only route that loads them -- `call_archaic_segments_for_subject` calls `ensure_ancestry_asset` for each before reading (haplogroup.rs:3596). They are in the manifest, so they are sha256-verified on fetch like every other asset. - First-run seeding is a directory scan (`seed_assets_from`), so an asset that is not bundled is simply not seeded rather than erroring. Also documents the trap in packaging-and-release.md, since the next asset will hit it: silence is not a default, it is the CI-only bundle. Co-Authored-By: Claude Opus 5 (1M context) --- documents/design/packaging-and-release.md | 14 ++++++++++++++ packaging/stage-assets.sh | 17 +++++++++++++++-- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/documents/design/packaging-and-release.md b/documents/design/packaging-and-release.md index 871a7ad..ab51ac2 100644 --- a/documents/design/packaging-and-release.md +++ b/documents/design/packaging-and-release.md @@ -241,9 +241,23 @@ pipeline asset (sizes measured on disk) split the rest by size × volatility: | Y/mt haplotrees | 60–127 MB | High (~weekly) | **Runtime fetch + conditional GET** | ETag/`If-None-Match` → 304 (implemented 2026-07-10); optionally Cloudflare-free in front of the AppView. **Not** the paid S3 CDN. | | chrY callable/cohort masks | ~1.8 MB | Rare | **Installer bundle** (git `assets/masks/` → seed) | Small, done. | | Ancestry/IBD `.bin` | ~160 MB | Rare | **Installer bundle** (GitHub release → stage → seed) | Decision A, done. | +| **Archaic Tier A** (`archaic_markers_`, `archaic_marker_dist_`) | 33 MB | Rare | **Installer bundle → seed** | Reachable by every user *including from chip data*, so it must work with no network. | +| **Archaic Tier B** (`archaic_outgroup_af_`, `archaic_classify_`, `archaic_callable_`) | 72 MB | Rare | **On demand** (`ensure_ancestry_asset`) | Segment calling needs WGS *and* a completed hours-long de-novo calling pass, so very few users reach it. Decided 2026-07-30 — see below. | | **STR HipSTR reference** | 20 MB (GRCh38) / 0.2 MB (CHM13) | Static | **Installer bundle → seed** (release-hosted, wired 2026-07-11) | Small, static; delivery path added below. | | **Y-SNP dictionary** (`dictionary.tsv`) | ~208 MB | Medium (~weekly YBrowse) | **Lazy download from GitHub release, sha256-verified** | Too big to bundle, too volatile; had *no* delivery path. | +**Archaic asset split — decided 2026-07-30, after it shipped wrong.** The archaic design's M1 said to +check Asset 1's size against `ON_DEMAND_PREFIXES` "before deciding"; no decision was ever recorded, so +`archaic_*` appeared in **neither** `PATTERNS` nor `ON_DEMAND_PREFIXES`. Those two lists are the same +decision written once per source mode — local-dir mode bundles what `PATTERNS` names, release mode +bundles everything the manifest names *except* `ON_DEMAND_PREFIXES` — so an asset missing from both is +**bundled in CI and absent in dev builds**. All five archaic assets (105.3 MB) therefore shipped in +`v0.1.0-alpha.14`, whose installers came out ~60 MB larger than `alpha.13`'s (`.dmg` 136→199 MB, +`.exe` 105→163 MB) with no one having chosen that. + +Both lists now name every archaic asset explicitly: Tier A bundled, Tier B on demand. **When adding an +asset, put it in one list or the other — silence is not a default, it is the CI-only bundle.** + **Cost stance (creative GitHub):** GitHub **Releases** is the free asset host for app-built data (ancestry `.bin`, STR ref, Y-SNP dict) — public repo, tokenless HTTPS, no egress bill. Do **not** wire the designed-but-unused S3/CloudFront path (`scripts/ancestry-panel/06_publish_cdn.sh`); it's diff --git a/packaging/stage-assets.sh b/packaging/stage-assets.sh index 481de9a..7dca1e4 100755 --- a/packaging/stage-assets.sh +++ b/packaging/stage-assets.sh @@ -48,10 +48,21 @@ ASSET_BUILD="${NAVIGATOR_ASSET_BUILD:-chm13v2.0}" # needs it, so bundling it would grow every download for a feature many users never open. # Local-dir mode omits it by simply not listing it in PATTERNS; release mode has to filter, because # it fetches whatever the manifest names. -ON_DEMAND_PREFIXES="ancestry_haps_" +# +# The three Tier B ARCHAIC assets follow the same rule, and for a stronger reason than the painter: +# archaic *segment* calling needs whole-genome data AND an hours-long de-novo calling pass to have +# already been run, so the number of users who reach it is very small, while the three assets total +# 72 MB. Tier A (the marker panel + percentile reference, 33 MB) stays in the bundle — it is the +# report every user can reach, including from chip data, so it must work with no network. +ON_DEMAND_PREFIXES="ancestry_haps_ archaic_outgroup_af_ archaic_classify_ archaic_callable_" -# The full Option-A bundle: ancestry panels/PCA/freqs + manifest + genetic map + IBD panel. +# The full Option-A bundle: ancestry panels/PCA/freqs + manifest + genetic map + IBD panel + the +# Tier A archaic assets. # NOTE: keep in step with ON_DEMAND_PREFIXES above — anything omitted here must be downloadable. +# The two lists are the SAME decision written twice, once per source mode: local-dir mode bundles +# exactly what PATTERNS names, while release mode bundles everything the manifest names EXCEPT +# ON_DEMAND_PREFIXES. An asset absent from both lists is bundled in CI and missing in dev builds — +# which is what happened to the archaic assets in v0.1.0-alpha.14 and cost ~60 MB per installer. PATTERNS=( "ancestry_panel_"*.bin "ancestry_pca_"*.bin @@ -61,6 +72,8 @@ PATTERNS=( "ancestry_manifest_"*.json "genetic_map_"*.bin "ibd_panel_"*.bin + "archaic_markers_"*.bin + "archaic_marker_dist_"*.bin ) sha256_of() {