Skip to content

feat: judge originator rankings against gold lineages - #46

Merged
hedmana merged 3 commits into
mainfrom
feat/lineage-evaluation
Sep 29, 2026
Merged

hedmana merged 3 commits into
mainfrom
feat/lineage-evaluation

Conversation

@hedmana

@hedmana hedmana commented Sep 29, 2026

Copy link
Copy Markdown
Owner

Closes #45

What

  • gnn.lineages judges the rankings a reader sees. It reads fields/gold/<field>.yaml: papers, and the earlier works each grew from, as lists of OpenAlex ids, any of which counts. For each paper it ranks the ancestry with the learned scorer, the same walk with equal weights, and every baseline in graph.score. It reports:

    • Recall@10 and nDCG@10, over the gold originators reachable from the paper.
    • Coverage, reported apart: a work the crawl never reached counts against coverage, not against a scorer.
    • The learned top 10's overlap across training seeds (3 by default).
    • Every scorer's top-10 overlap when 10% of citations are dropped.

    It writes lineages.json beside the snapshot.

  • originators.flow is the damped walk split out of learned_flow, so the equal-weight ablation runs the same walk. learned_flow's behaviour is unchanged.

  • The gold set is not in this PR. The draft is being verified and will land on its own.

Why

Everything measured so far is the link-prediction proxy. On the draft gold set (18 papers, 94 originators) and the served snapshot:

Scorer Recall@10 nDCG@10 Overlap, 10% dropped
learned 0.495 0.484 0.844
equal-weight walk 0.444 0.395 0.856
time_decayed_pagerank 0.432 0.380 0.833
in_degree 0.127 0.076 0.833
gateway 0.074 0.097 0.822
path_weight 0.000 0.000 0.939

Coverage is 90/94, and the learned top 10 overlaps 96% across three seeds. The learned ranking leads, and the model's weights are worth +0.05 recall and +0.09 nDCG over the same walk with equal weights. path_weight ranks 1980s foundations first, the failure time decay exists to prevent. These numbers are provisional until the gold set is verified.

Verify

uv run pytest tests/test_lineages.py tests/test_originators.py
uv run python -m gnn.lineages --field gnn --gold path/to/gold.yaml   # about 3 minutes

Generated with Claude Code

@hedmana
hedmana merged commit a29ab6d into main Sep 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge originator rankings against gold lineages

1 participant