Skip to content

Decide the archaic asset staging split; both modes were disagreeing silently #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JamesKane merged 1 commit into from
Jul 30, 2026
Merged

Decide the archaic asset staging split; both modes were disagreeing silently #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions documents/design/packaging-and-release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,9 +241,23 @@ pipeline asset (sizes measured on disk) split the rest by size × volatility:
| Y/mt haplotrees | 60–127 MB | High (~weekly) | **Runtime fetch + conditional GET** | ETag/`If-None-Match` → 304 (implemented 2026-07-10); optionally Cloudflare-free in front of the AppView. **Not** the paid S3 CDN. |
| chrY callable/cohort masks | ~1.8 MB | Rare | **Installer bundle** (git `assets/masks/` → seed) | Small, done. |
| Ancestry/IBD `.bin` | ~160 MB | Rare | **Installer bundle** (GitHub release → stage → seed) | Decision A, done. |
| **Archaic Tier A** (`archaic_markers_`, `archaic_marker_dist_`) | 33 MB | Rare | **Installer bundle → seed** | Reachable by every user *including from chip data*, so it must work with no network. |
| **Archaic Tier B** (`archaic_outgroup_af_`, `archaic_classify_`, `archaic_callable_`) | 72 MB | Rare | **On demand** (`ensure_ancestry_asset`) | Segment calling needs WGS *and* a completed hours-long de-novo calling pass, so very few users reach it. Decided 2026-07-30 — see below. |
| **STR HipSTR reference** | 20 MB (GRCh38) / 0.2 MB (CHM13) | Static | **Installer bundle → seed** (release-hosted, wired 2026-07-11) | Small, static; delivery path added below. |
| **Y-SNP dictionary** (`dictionary.tsv`) | ~208 MB | Medium (~weekly YBrowse) | **Lazy download from GitHub release, sha256-verified** | Too big to bundle, too volatile; had *no* delivery path. |

**Archaic asset split — decided 2026-07-30, after it shipped wrong.** The archaic design's M1 said to
check Asset 1's size against `ON_DEMAND_PREFIXES` "before deciding"; no decision was ever recorded, so
`archaic_*` appeared in **neither** `PATTERNS` nor `ON_DEMAND_PREFIXES`. Those two lists are the same
decision written once per source mode — local-dir mode bundles what `PATTERNS` names, release mode
bundles everything the manifest names *except* `ON_DEMAND_PREFIXES` — so an asset missing from both is
**bundled in CI and absent in dev builds**. All five archaic assets (105.3 MB) therefore shipped in
`v0.1.0-alpha.14`, whose installers came out ~60 MB larger than `alpha.13`'s (`.dmg` 136→199 MB,
`.exe` 105→163 MB) with no one having chosen that.

Both lists now name every archaic asset explicitly: Tier A bundled, Tier B on demand. **When adding an
asset, put it in one list or the other — silence is not a default, it is the CI-only bundle.**

**Cost stance (creative GitHub):** GitHub **Releases** is the free asset host for app-built data
(ancestry `.bin`, STR ref, Y-SNP dict) — public repo, tokenless HTTPS, no egress bill. Do **not**
wire the designed-but-unused S3/CloudFront path (`scripts/ancestry-panel/06_publish_cdn.sh`); it's
Expand Down
17 changes: 15 additions & 2 deletions packaging/stage-assets.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,21 @@ ASSET_BUILD="${NAVIGATOR_ASSET_BUILD:-chm13v2.0}"
# needs it, so bundling it would grow every download for a feature many users never open.
# Local-dir mode omits it by simply not listing it in PATTERNS; release mode has to filter, because
# it fetches whatever the manifest names.
ON_DEMAND_PREFIXES="ancestry_haps_"
#
# The three Tier B ARCHAIC assets follow the same rule, and for a stronger reason than the painter:
# archaic *segment* calling needs whole-genome data AND an hours-long de-novo calling pass to have
# already been run, so the number of users who reach it is very small, while the three assets total
# 72 MB. Tier A (the marker panel + percentile reference, 33 MB) stays in the bundle — it is the
# report every user can reach, including from chip data, so it must work with no network.
ON_DEMAND_PREFIXES="ancestry_haps_ archaic_outgroup_af_ archaic_classify_ archaic_callable_"

# The full Option-A bundle: ancestry panels/PCA/freqs + manifest + genetic map + IBD panel.
# The full Option-A bundle: ancestry panels/PCA/freqs + manifest + genetic map + IBD panel + the
# Tier A archaic assets.
# NOTE: keep in step with ON_DEMAND_PREFIXES above — anything omitted here must be downloadable.
# The two lists are the SAME decision written twice, once per source mode: local-dir mode bundles
# exactly what PATTERNS names, while release mode bundles everything the manifest names EXCEPT
# ON_DEMAND_PREFIXES. An asset absent from both lists is bundled in CI and missing in dev builds —
# which is what happened to the archaic assets in v0.1.0-alpha.14 and cost ~60 MB per installer.
PATTERNS=(
"ancestry_panel_"*.bin
"ancestry_pca_"*.bin
Expand All @@ -61,6 +72,8 @@ PATTERNS=(
"ancestry_manifest_"*.json
"genetic_map_"*.bin
"ibd_panel_"*.bin
"archaic_markers_"*.bin
"archaic_marker_dist_"*.bin
)

sha256_of() {
Expand Down
Loading