Feat/ancestral origin icicle - #31
Merged
Merged
Conversation
`/api/v1/ibd/attest` gates on `owns_sample(attester_did, claimed_sample)`, but a self-publishing Edge client has no way to learn its server-side `core.biosample.sample_guid` — the suggestions payload returned only the *candidate's* guid, so Navigator could never fill in `claimed_sample` and the attest endpoint was unreachable from the edge. `suggestions_for_did` already joins on `ms.target_sample_guid` (that is how the per-DID scope is enforced), so surfacing it costs nothing and leaks nothing: the caller owns that sample by construction. `suggested_sample_guid` supplies the `counterpart_sample` of the same report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The de-novo loader reuses a catalog row only when it matches on
`coordinates @> {'hs1': ...}`, so markers whose hs1 coordinate had not been
lifted yet missed the match and got a fresh hs1-only row — and that row is what
`tree.haplogroup_variant` points at. `variant-name-reconcile` later adopted the
marker's name onto the branch row but never touches `coordinates`, leaving the
tree with correct names and, on the 2026-08-04 prod dump, GRCh38 for only 44,181
of 203,983 branch SNPs (21.7%).
The Navigator places each source in its native build with no liftover
(`place_y_consensus_decodingus`), and `parse_decodingus_json(json, build_key)`
drops every locus lacking that build's coordinate — so a GRCh38 subject saw
3,413 of 11,421 Y nodes. All 33 backbone nodes survived; what vanished was the
terminal tree (node visibility 81% at depth 0-10 → 21% at 31-40), so GRCh38
subjects placed plausibly but shallow rather than failing outright.
Copy GRCh38/GRCh37 from the marker's catalog row (same canonical_name AND
identical hs1 site + alleles) rather than chain-lifting: the catalog row holds
YBrowse's own values, and a lift would re-derive them and can mismap in the
inverted / ampliconic Y blocks. All 132,183 candidate rows agree with their twin
on position and alleles (0 swapped), so the copy is exact. Rows whose twin
disagrees are not filled — the match is enforced in the join.
Verified against a restored copy of the 2026-08-04 prod dump and against
decodingus_cutover: 130,553 rows filled in 34s, GRCh38 44,181 → 174,734, node
visibility 3,413 → 11,104/11,421, flat across every depth band. Idempotent — a
re-run fills 0 and leaves tree_revision alone (the ~60 MB tree payload should
not be invalidated for a no-op).
Residual ~27.6k rows have no named twin and need variant-coord-lift, which
requires the reverse hs1->GRCh38 chain staged to establish its pivot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The public tree answers where a clade sits phylogenetically and, in the
"Geography & Time" panel, where its samples were collected. Neither answers what
a surname project actually asks: as this branch splits, where do the lines go?
ytree.net answers it by putting geography onto the phylogeny itself. This adds
that view — the Big Tree's top-down icicle, containment carrying descent, with
each band filled by where its men's most distant known ancestors came from.
The AppView had nothing to draw it from. Its only locality datum is
`core.specimen_donor.geocoord`, and of 9,642 placed Y samples 1,380 carry one —
all of them ancient or academic. The 7,882 `cohort=bigy` D2C tips, which are the
entire genealogical era, have 3 between them. So this also adds the substrate: a
`com.decodingus.atmosphere.ancestralOrigin` lexicon, mirrored to
`fed.ancestral_origin` (migration 0074) by the existing Jetstream consumer.
MDKA as publishable data is not new policy — `biosample-identifier-dedup.md`
already records it as "genealogical context, not PII". What is new is that the
AppView now *enforces* that rather than asserting it. Five gates run at ingest
and REJECT the record, never store-and-hide, because a row that exists is a row
some future read path can leak: a single-token surname (particles allowed, so
`van der Berg` survives and `Thomas Michael Kane` does not), `birthYear <= 1900`,
country-only when no birth year establishes the ancestor is long dead,
coordinates re-coarsened to ~1 km whatever the client sent, and a join key that
is never rendered. The bulk-load exclusion in `import_kit_identifiers.rs` stands:
origins enter only when a PDS publishes them, so this ships dark until the
Navigator half lands.
The two migration headers that say otherwise (`0012_fed_reporting` here,
`0030_mdka` in Navigator) are deliberately NOT edited — both repos run
`sqlx::migrate!`, which checksums applied migrations, so changing even a comment
would fail every existing database. `proposals/ancestral-origin-icicle.md` §2 is
the amendment of record and 0074's header points at it. The D4 assertion-store
rail rejecting MDKA_IS is untouched: it governs assertions about a *living*
subject, which is the distinction that keeps the two apart.
Resolution runs through `core.biosample_identifier`, not the at-uri. Zero placed
samples carry an at-uri — the tips were bulk-loaded, not federated — while 7,548
carry an FTDNA row from migrations 0059/0060, which exist precisely to match a
re-published donor to its existing biosample. Both paths are unioned so at-uri
works as federation grows.
Three things came from measuring rather than reasoning:
* Place normalization was validated against all 3,356 real MDKA strings, which
caught what hand-picked cases missed. US ZIPs went unstripped, making dozens
of singleton "admins" (`Va 24521`, `Wv 26801`) that are all one state; the
country table was too short; and a parenthetical qualifier dropped the row
entirely. 705 distinct admin strings fold to 457, with 0 unresolvable and 6
no-country out of 3,356.
* A band spans its PARENT's TMRCA to its own, not its own `formed_ybp`. The
obvious choice is wrong: the two are independent point estimates under no
monotonicity constraint, agreeing on 898 of 10,252 edges while 4,243 (41%)
have the child forming before its parent's split. Rendering the real tree
put `R-A13318` at exactly its parent `R-S764`'s y. Parent-TMRCA → own-TMRCA
has zero inversions, so containment holds by construction.
* Nothing is dropped silently. A sample on a de-novo node used to contribute
to nothing, so every band above it understated itself; it now climbs to the
nearest named ancestor as sample tips already do. Branches with no origin
beneath them are pruned — unpruned, R-S764 drew 175 bands across 7,944px to
show 10 origins; pruned it is 37 in 768px — and the count, the placed total,
and the samples with no published origin are all stated on the page.
Colour is the validated 8-slot categorical palette, fixed order, never cycled; a
ninth locality folds into a reserved neutral that also carries "no locality
recorded", since an absence is not an identity. Both modes pass the checker;
light mode's contrast warning is met by the always-on band labels and the table
view.
Tests: 14 normalizer, 6 ingest-gate, 18 layout. Suites green (du-db 52, du-jobs
46, du-web 93). Verified end to end against the dev DB with synthetic records on
real FTDNA kits, which resolved to real placed nodes; those rows were removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eir boxes The icicle ships dark — `import_kit_identifiers.rs` reserves MDKA for records a PDS publishes, and the Navigator publisher is not built — so there was no honest way to put the view in front of a reviewer. `seed-ancestral-origins.sql` fabricates a plausible cohort against REAL tree placements: 708 rows over three era-gated clades (R-DF85, R-S764, R-Z3000), every row stamped `did = 'did:plc:preview'` so the set removes with one delete and can never be mistaken for a contributor's record. The mix is deliberately awkward rather than tidy, because the tidy version hides the cases that matter: more than eight distinct counties (so the fold into the reserved "Other" slot is exercised), US diaspora beside Irish counties, country-only rows, and men with no locality at all. Rows are written already conformant to the ingest gates — single-token surnames, birth years at or before 1900, 2dp coordinates, and no place or coordinate where there is no birth year — so the preview shows what ingest would actually have kept rather than a state it would have rejected. Seeding it immediately exposed a bug that the layout tests could not: the boxes are sized by the phylogeny, not by the text. A tip is at most 74px and holds about 13 characters, while `Sullivan · Co. Limerick` is 23, so labels ran straight through their neighbours and the tip row was unreadable. `fit()` now truncates band and tip labels to their box with an ellipsis, and the unabbreviated form moves to the hover title so nothing is lost. The rectangle assertions all passed throughout — this was only ever visible by rendering it. Also recorded from the same run, unfixed: with dense data pruning stops helping. R-DF85 draws 266 bands across 11,220px because 278 of its 283 samples carry an origin, and the page scrolls horizontally like the Big Tree and FTDNA's block tree do. That is inherent to one box per man at a legible width, not a defect, but it is the shape a width-reduction pass would target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
R-DF85 drew 266 branches across an 11,220px canvas — six screens of horizontal scrolling for a chart nobody could read. Canvas width is driven by the number of leaf branches, so pruning empty ones only helps where data is sparse; on a well-covered clade (278 of 283 samples carry an origin) it does nothing. Three changes, all found by looking at the rendered page rather than the tests. A DEPTH BOUND, defaulting to 4 levels with a selector (1-8). This is a legibility bound, not a data one: branches past the depth are marked `hidden` rather than dropped, which routes them through the same path that already handles de-novo nodes — their men are attributed to the nearest drawn ancestor. So every band's composition is identical at every depth and only the visible branching changes, which `folding_by_depth_preserves_composition_exactly` pins. R-DF85 goes 11,220px → 3,498px at the default, 2,016px at depth 2. Folded bands carry a "+" and say so on hover, so "this branch is simple" is never confused with "you are not being shown its shape". TIPS TOO NARROW TO LABEL ARE DROPPED AND COUNTED. Men share their band's width, so a band holding forty of them produced forty 8px slivers that hid the composition bar above them rather than adding anything. Below 26px the box goes and the man is counted instead — reported on the page, never silently. "NO LOCALITY RECORDED" IS NOW DRAWN. It was counted, legended, and then left as bare band background, so the chart disagreed with its own legend and a branch of unrecorded men looked like a branch with fewer men. It is now a real segment, always last so absence sits at the same end of every bar. That exposed a second problem: slot 0 carries both "Other" (localities past the eight palette slots) and "no locality recorded", and the legend called both of them the latter. They share a colour but not a meaning, so `Segment::unknown` now separates them in the legend, the tooltips and the table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…to the men Two corrections to what a block means. A BLOCK IS ITS SNPs. The branch's phylogenetically equivalent mutations are unordered — nothing separates them — so the list *is* the block, exactly as the Big Tree draws it. They were missing entirely; a block was a bare rectangle. `variant_names_for` fetches them for the whole window in one query rather than one per branch, and they flow into as many columns as the block's width allows and as many rows as its height allows. The block is never grown to fit the list: its height is elapsed time and has to stay on the shared axis, so what does not fit is reported as "+N" on a line of its own. BLOCKS ARE NO LONGER TINTED BY ORIGIN. Colouring a clade by the composition of its descendants asserted something the data does not support — a branch has no locality, only the men standing on it do, and a modal-origin tint reads as a claim about the whole lineage. The colour now lives exactly where the claim does: on each man's box, keyed to his own most distant known ancestor. The legend and table still carry the composition, which is what explains those colours, so `Segment` became a tally rather than a drawn mark and lost its geometry. Two layout bugs that only rendering showed. A leaf block is exactly LEAF_W wide, which is narrower than one preferred SNP column, so flooring the column count gave it zero columns and dropped its SNPs — the common case, not an edge one; a block now always gets at least one column, sized to what it actually has. And the SNP list was anchored to the block's padding rather than to the name's baseline, so every block opened with its name and first SNP overprinted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A block with 16 equivalent SNPs was drawing as an 18px sliver. Sizing blocks by
elapsed years does not survive contact with the data, and both obvious forms of
it fail:
* a node's own formed_ybp → its own tmrca_ybp draws children ON TOP of their
parents — the two are independent point estimates under no monotonicity
constraint, agreeing on 898 of 10,252 edges while 4,243 (41%) have the child
forming before its parent's split (fixed earlier, in 4ffc3b8);
* parent TMRCA → own TMRCA is monotone, so containment held — but it is
degenerate. `formed_ybp == tmrca_ybp` on 41% of terminal branches and 26.5%
of internal ones, collapsing those branches to a point. On R-DF85 at depth 4
that left 30 of 75 blocks unable to show a single one of their SNPs:
R-BY18328 got 3px of span for 9 mutations, R-BY170664 16 SNPs in an 18px
sliver, and a quarter of all blocks sat pinned at the minimum height.
So the block is sized to its SNPs — one line each, nothing elided — which is
what BACKLOG's own block-tree convention already says and what the Big Tree
does. The list is the block; truncating it shortens the box, and a shortened box
misreports how long the branch ran unbroken. Vertical position becomes
cumulative, and the gutter rules off in mutations rather than calendar years.
This loses nothing as a time axis. Mutations accrue at a roughly steady rate, and
measured on this very tree branch length tracks SNP count at r = 0.975, about 69
years per mutation — better behaved than the per-branch estimate it replaces,
which is missing or degenerate exactly where a block most needs a height. Ages
keep their two real jobs: gating the view to the genealogical era, and labelling
each block.
Result on R-DF85 at depth 4: every one of 468 SNPs drawn, 0 blocks hiding any,
and the canvas got SHORTER (706px → 439px) because blocks now take the height
their content needs instead of whatever a sparse time axis handed them. The "+N
did not fit" marker is gone with the problem it reported.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rigins Two pieces of scaffolding outlived what they were for. THE CROSS-HATCH marked branches with no age estimate. It was justified only because height *was* the age span — an unmeasured branch must not read as a short one. Height is SNP count now (79305f9), so the hatch distinguishes nothing and is just noise on 16% of blocks. Gone, along with the `dated` flag behind it. BRANCHES WITH NO PUBLISHED ORIGIN BENEATH THEM were pruned away. That was how the 11,220px canvas got tamed before the depth bound existed; the depth bound does that job now, and pruning is both redundant and wrong. Origins are an overlay on the tree, not a filter of it — a branch with no locality data is still part of the clade's shape, and dropping it misrepresents the phylogeny to make a sparse overlay look dense. Removing the prune surfaced the tip row's own version of the same confusion. Men were packed into a single row under their branch, so most became slivers narrower than their labels and were dropped and counted instead — 219 of 278 on R-DF85, and with the block tint gone they were then shown nowhere at all. Men now wrap into rows, sized for a readable box first and narrowed only when the branch demands it. But wrapping alone made one depth-4 boundary block absorb 179 folded men and stack them 90 rows deep, a tip row taller than the tree above it. So a man gets a box only where his own branch is drawn: folded upward he still counts in his ancestor's composition, the fold marker is the affordance, and drilling in draws him where he belongs. R-DF85 at depth 4: 76 branches (was 75 — the pruned one is back), all 471 SNPs, no hatch, 64 men drawn and every one legible, canvas 459px. At depth 6 the same view resolves 108 men, which is what drilling in is meant to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, one of them mine to begin with. THE BLOCK HEADERS WERE ILLEGIBLE ON A DARK-MODE DESKTOP. The icicle carried a `prefers-color-scheme: dark` block, but the site itself is light-only — so on a dark OS the SVG surface went dark while the Bootstrap-derived block fills stayed light, and every branch name became dark text with a dark halo on a light block. Keying off the OS was wrong in the first place: a chart embedded in a page has to follow that page. The dark values now hang off `[data-bs-theme="dark"]` alone, which is what the app would set if it ever gains a dark theme, and `--surface-1` defaults to `--bs-body-bg` so it tracks whatever the page is. The label halo also now uses the block fill rather than the canvas, since the block is what sits behind the text. THE TYPE WAS TOO SMALL — 9px SNP names, 10px branch names. Names go to 12px bold, SNPs and men's labels to 11px, the ruler to 10px, and the geometry is now derived from those sizes rather than hard-coded beside them: line heights, tip height, minimum tip width and the leaf block width all follow the font constants, so changing the type cannot silently break the layout again. A leaf block widens 74px → 90px, which is what a 12-character SNP name (`14405732-C-T`) actually needs at 11px. R-DF85 at depth 4 costs 3,576px → 4,296px of width for it. That is the trade: legible at normal viewing distance instead of technically-present. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The view existed but was reachable only by typing its URL. Three ways in now, each at the altitude it belongs to. PER CLADE — a button in the tree's SNP sidebar, beside the sample-map panel that answers the neighbouring question. It navigates rather than swapping a fragment: the icicle is a whole view, not a sidebar widget. PER LINEAGE — `/ytree/origins` and `/mtree/origins`, the Tools menu entry. These open on the tree's default root, which is always older than the era gate, so they render the signpost — and that turns out to be the right landing page rather than a dead end: it states the constraint and immediately offers the clades young enough to have origins. For that to be useful the signpost had to stop listing whatever the tree walk happened to return. It now ranks by how many men a clade holds and drops the ones holding none, so `/ytree/origins` opens on R-DF85 (283), R-S673 (239), R-S764 (232) rather than the alphabetically-first branches. 5,113 clades qualify under the Y root; 60 are shown and the page says so, because a capped list that looks complete is worse than one that admits its cap. Each entry carries its sample count and TMRCA, which is what a reader picks on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.