feat: judge originator rankings against gold lineages

README.md

@@ -71,6 +71,7 @@ uv run python -m graph.build --field gnn        # clean it into a dated snapshot
 uv run python -m gnn.dataset --field gnn        # cut the temporal link-prediction split
 uv run python -m gnn.train --field gnn          # fit, and score on citations it never saw
 uv run python -m gnn.evaluate --field gnn       # score it beside the heuristics it has to beat
+uv run python -m gnn.lineages --field gnn       # judge the rankings against fields/gold/gnn.yaml
 uv run python -m llm.narrative --field gnn --target W2519887557
 ```
 

src/gnn/PLAN.md

@@ -53,7 +53,18 @@ One PR each, named for what it does.
   - Hashed title words: -0.02, overfits.
   - `topic_id`: +0.06, but it leaks. OpenAlex assigns topics from citations as of the crawl, the reason `cited_by_count` is excluded.
   - Held-out share, learning rate, depth, width, dropout, weight decay, and MRR instead of AP for checkpoint selection: nothing beyond noise.
-- [ ] **Originator ranking evaluation.** Gold lineages for 20-30 targets in `fields/gold/gnn.yaml` (not `fields/*.yaml`, which the API lists as fields), from survey history sections. nDCG@10 and Recall@10 vs all baselines. Ablation: `learned_flow` with uniform edge weights. Stability: top-10 overlap across seeds and 10% edge dropout.
+- [ ] **Originator ranking evaluation.** `gnn.lineages` is built: Recall@10 and nDCG@10 over reachable gold originators for the learned ranking, the same walk with equal weights, and every baseline; coverage apart; top-10 overlap across 3 training seeds and with 10% of citations dropped. Waiting on the gold set, `fields/gold/gnn.yaml` (not `fields/*.yaml`, which the API lists as fields): 18 targets drafted, being verified. On the draft, served snapshot:
+
+  | Scorer | Recall@10 | nDCG@10 | Overlap, 10% dropped |
+  |---|---|---|---|
+  | learned | 0.495 | 0.484 | 0.844 |
+  | equal-weight walk | 0.444 | 0.395 | 0.856 |
+  | time_decayed_pagerank | 0.432 | 0.380 | 0.833 |
+  | in_degree | 0.127 | 0.076 | 0.833 |
+  | gateway | 0.074 | 0.097 | 0.822 |
+  | path_weight | 0.000 | 0.000 | 0.939 |
+
+  Coverage 90/94; learned top-10 overlap across seeds 0.961. `path_weight` ranks 1980s foundations first, the failure time decay exists to prevent.
 - [ ] **Citation influence labels.** Semantic Scholar `isInfluential` per citation; per-paper AUC of the decoder on influential vs incidental references.
 - [ ] **Influence-aligned objective.** Fine-tune the decoder listwise on influential references, link prediction as pretraining. Validate on the gold set.
 - [x] **Crawl expansion.** Uncapped backward crawl, plus `forward_depth: 1`: the in-topic works citing the seeds. 2789 works and 15k citations became 9249 and 104k. GAT is seeded by its primary record. Cleaning re-dates 262 works OpenAlex dates years after their citers, and a year heavier than its split's share no longer empties the split after it. Gold originators reachable from their targets: 88/94 to 90/94. The other four are references OpenAlex holds under dead ids (about 12% of all references), among them GCN's to Bruna, Henaff and Planetoid.
@@ -63,5 +74,5 @@ Optional: blinded LLM pairwise judge, trusted only after it agrees with the gold
 ## Open decisions
 
 - Semantic Scholar as a second data source: citation influence labels, and the references OpenAlex holds under dead ids.
-- Who verifies the gold set (originator ranking evaluation).
+- Gold set verification, then commit `fields/gold/gnn.yaml` and rerun `gnn.lineages`.
 - Abstract embeddings need a new dependency; deferred until originator ranking evaluation shows text is the bottleneck.

src/gnn/lineages.py

@@ -0,0 +1,247 @@
+"""Judge originator rankings against lineages a person has checked.
+
+Link prediction says whether the model understands citations. It does not say
+whether the ranking a reader sees names the works that made a paper possible.
+That takes an answer key, `fields/gold/<field>.yaml`: papers, and for each the
+earlier works it grew from. Every scorer ranks each paper's ancestry and is
+judged on how many of those works reach its top ten, and how high.
+
+A gold originator the crawl never reached, or reached by no path from its paper,
+cannot be ranked by anything. It counts against coverage, reported on its own,
+rather than against a scorer.
+
+Two further checks ask whether a ranking can be trusted at all. The learned one
+should not change with the seed its model was trained from, and no ranking
+should change much when a tenth of the citations go missing, since OpenAlex
+already loses about that many.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import math
+import random
+from itertools import combinations
+from pathlib import Path
+
+import networkx as nx
+import yaml
+from pydantic import BaseModel
+from torch import Tensor
+
+from gnn.dataset import build_dataset
+from gnn.model import LinkPredictor
+from gnn.originators import embed, flow, learned_flow
+from gnn.train import TrainConfig, load_model, train
+from graph.build import is_missing, latest_snapshot, load_snapshot
+from graph.config import FIELDS_DIR
+from graph.crawler import DATA_DIR
+from graph.score import SCORERS, acyclic, ancestry, rank
+
+logger = logging.getLogger(__name__)
+
+GOLD_DIR = FIELDS_DIR / "gold"
+
+
+class Lineage(BaseModel):
+    target: str
+    # One entry per work, as every OpenAlex id it goes by: any of them counts.
+    originators: list[list[str]]
+
+
+class Gold(BaseModel):
+    lineages: list[Lineage]
+
+
+def load_gold(path: Path) -> list[Lineage]:
+    if not path.is_file():
+        raise FileNotFoundError(f"no gold lineages at {path}")
+    return Gold.model_validate(yaml.safe_load(path.read_text())).lineages
+
+
+def rankings(
+    graph: nx.DiGraph,
+    target: str,
+    model: LinkPredictor,
+    embedding: tuple[dict[str, int], Tensor],
+    top: int,
+) -> dict[str, list[str]]:
+    """Every scorer's top works for one paper: the model, its walk unweighted, the baselines."""
+    scores = {
+        "learned": learned_flow(graph, target, model, embedding=embedding),
+        "uniform_flow": flow(acyclic(ancestry(graph, target)), target),
+        **{name: scorer(graph, target) for name, scorer in SCORERS.items()},
+    }
+    return {name: [node for node, _ in rank(ranked, top)] for name, ranked in scores.items()}
+
+
+def judge(ranking: list[str], findable: list[list[str]], top: int) -> tuple[float, float]:
+    """Recall and nDCG of a top-`top` ranking, over the gold originators it could have found."""
+    if not findable:
+        return math.nan, math.nan
+    work = {work_id: order for order, ids in enumerate(findable) for work_id in ids}
+    found: set[int] = set()
+    gain = 0.0
+    for position, node in enumerate(ranking, start=1):
+        if (order := work.get(node)) is not None and order not in found:
+            found.add(order)
+            gain += 1 / math.log2(position + 1)
+    ideal = sum(1 / math.log2(position + 1) for position in range(1, min(len(findable), top) + 1))
+    return len(found) / len(findable), gain / ideal
+
+
+def _overlap(left: list[str], right: list[str]) -> float:
+    return len(set(left) & set(right)) / len(left) if left else math.nan
+
+
+def _thinned(graph: nx.DiGraph, share: float, seed: int) -> nx.DiGraph:
+    """The graph with a random `share` of its citations gone."""
+    chance = random.Random(seed)
+    thinned = nx.DiGraph()
+    thinned.add_nodes_from(graph.nodes(data=True))
+    thinned.add_edges_from(edge for edge in graph.edges if chance.random() >= share)
+    return thinned
+
+
+def _mean(values) -> float:
+    kept = [value for value in values if not math.isnan(value)]
+    return sum(kept) / len(kept) if kept else math.nan
+
+
+def assess(
+    graph: nx.DiGraph,
+    gold: list[Lineage],
+    model: LinkPredictor,
+    *,
+    top: int = 10,
+    seeds: int = 3,
+    dropout: float = 0.1,
+) -> dict:
+    """Score, cover and stress every scorer on every gold paper the graph holds."""
+    lineages = [lineage for lineage in gold if lineage.target in graph]
+    for lineage in gold:
+        if lineage.target not in graph:
+            logger.warning("gold target %s is not in the graph", lineage.target)
+
+    embedding = embed(graph, model)
+    thinned = _thinned(graph, dropout, seed=0)
+    thinned_embedding = embed(thinned, model)
+    others = [
+        train(build_dataset(graph, seed=seed), config=TrainConfig(seed=seed))[0]
+        for seed in range(1, seeds)
+    ]
+    other_embeddings = [embed(graph, other) for other in others]
+
+    targets = []
+    for lineage in lineages:
+        reachable = set(ancestry(graph, lineage.target))
+        findable = [ids for ids in lineage.originators if reachable.intersection(ids)]
+        ranked = rankings(graph, lineage.target, model, embedding, top)
+        stressed = rankings(thinned, lineage.target, model, thinned_embedding, top)
+        learned = [ranked["learned"]] + [
+            [node for node, _ in rank(learned_flow(graph, lineage.target, other, embedding=e), top)]
+            for other, e in zip(others, other_embeddings, strict=True)
+        ]
+        title = graph.nodes[lineage.target].get("title")
+        targets.append(
+            {
+                "target": lineage.target,
+                "title": None if is_missing(title) else str(title),
+                "originators": len(lineage.originators),
+                "reachable": len(findable),
+                "seed_overlap": _mean(_overlap(a, b) for a, b in combinations(learned, 2)),
+                "scorers": {
+                    name: dict(
+                        zip(("recall", "ndcg"), judge(ranking, findable, top), strict=True),
+                        dropout_overlap=_overlap(ranking, stressed[name]),
+                        top=ranking,
+                    )
+                    for name, ranking in ranked.items()
+                },
+            }
+        )
+
+    names = targets[0]["scorers"] if targets else {}
+    reachable, originators = (sum(t[key] for t in targets) for key in ("reachable", "originators"))
+    return {
+        "top": top,
+        "targets_in_graph": f"{len(lineages)}/{len(gold)}",
+        "coverage": f"{reachable}/{originators}",
+        "seed_overlap": _mean(t["seed_overlap"] for t in targets),
+        "scorers": {
+            name: {
+                metric: _mean(t["scorers"][name][metric] for t in targets)
+                for metric in ("recall", "ndcg", "dropout_overlap")
+            }
+            for name in names
+        },
+        "targets": targets,
+    }
+
+
+def log_assessment(result: dict) -> None:
+    logger.info("%-22s %10s %10s %16s", "scorer", "recall", "ndcg", "dropout overlap")
+    for name, metrics in result["scorers"].items():
+        logger.info(
+            "%-22s %10.3f %10.3f %16.3f",
+            name,
+            metrics["recall"],
+            metrics["ndcg"],
+            metrics["dropout_overlap"],
+        )
+    logger.info("gold targets in graph      %s", result["targets_in_graph"])
+    logger.info("gold originators reachable %s", result["coverage"])
+    logger.info("learned top-%d overlap across seeds %.3f", result["top"], result["seed_overlap"])
+
+
+def assess_field(
+    field_name: str,
+    *,
+    data_dir: Path = DATA_DIR,
+    gold_path: Path | None = None,
+    top: int = 10,
+    seeds: int = 3,
+    dropout: float = 0.1,
+) -> tuple[Path, dict]:
+    """Judge a field's served snapshot and model, and write the verdict beside them."""
+    snapshot = latest_snapshot(data_dir, field_name)
+    result = assess(
+        load_snapshot(snapshot),
+        load_gold(gold_path or GOLD_DIR / f"{field_name}.yaml"),
+        load_model(snapshot),
+        top=top,
+        seeds=seeds,
+        dropout=dropout,
+    )
+    path = snapshot / "lineages.json"
+    path.write_text(json.dumps(result, indent=2))
+    return path, result
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--field", default="gnn", help="field config name under fields/")
+    parser.add_argument("--data-dir", type=Path, default=DATA_DIR)
+    parser.add_argument("--gold", type=Path, default=None, help="defaults to fields/gold/<field>")
+    parser.add_argument("--top", type=int, default=10)
+    parser.add_argument("--seeds", type=int, default=3, help="models trained to test stability")
+    parser.add_argument("--dropout", type=float, default=0.1, help="share of citations removed")
+    args = parser.parse_args(argv)
+
+    logging.basicConfig(level=logging.INFO, format="%(message)s")
+    path, result = assess_field(
+        args.field,
+        data_dir=args.data_dir,
+        gold_path=args.gold,
+        top=args.top,
+        seeds=args.seeds,
+        dropout=args.dropout,
+    )
+    log_assessment(result)
+    logger.info("verdict written to %s", path)
+
+
+if __name__ == "__main__":
+    main()

src/gnn/originators.py

@@ -64,33 +64,44 @@ def learned_flow(
     half_life: float = 10.0,
     embedding: tuple[dict[str, int], Tensor] | None = None,
 ) -> Scores:
-    """Weight of a damped walk backwards from `target`, steered by the model.
-
-    Exact rather than iterated: the lineage is acyclic, so one pass in
-    topological order settles every node's share.
-    """
+    """Weight of a damped walk backwards from `target`, steered by the model."""
     lineage = acyclic(ancestry(graph, target))
     position, z = embedding or embed(graph, model)
 
     edges = list(lineage.edges())
     plausibility = dict(
         zip(edges, citation_probability(model, z, position, edges).tolist(), strict=True)
     )
+    return flow(lineage, target, plausibility, damping=damping, half_life=half_life)
+
+
+def flow(
+    lineage: nx.DiGraph,
+    target: str,
+    weight: dict[tuple[str, str], float] | None = None,
+    *,
+    damping: float = 0.85,
+    half_life: float = 10.0,
+) -> Scores:
+    """The damped walk itself, splitting each step by `weight`, or evenly without one.
 
-    flow = dict.fromkeys(lineage, 0.0)
-    flow[target] = 1.0
+    Exact rather than iterated: the lineage is acyclic, so one pass in
+    topological order settles every node's share.
+    """
+    reached = dict.fromkeys(lineage, 0.0)
+    reached[target] = 1.0
     for node in nx.topological_sort(lineage):
         cited = list(lineage.successors(node))
-        shares = [plausibility[(node, work)] for work in cited]
+        shares = [1.0 if weight is None else weight[(node, work)] for work in cited]
         total = sum(shares)
         if not total:
             continue
         for work, share in zip(cited, shares, strict=True):
-            flow[work] += flow[node] * damping * share / total
+            reached[work] += reached[node] * damping * share / total
 
     return {
-        node: weight * 0.5 ** (year_gap(lineage, target, node) / half_life)
-        for node, weight in flow.items()
+        node: share * 0.5 ** (year_gap(lineage, target, node) / half_life)
+        for node, share in reached.items()
         if node != target
     }