Skip to content

feat: recover citations OpenAlex lost, from Semantic Scholar - #50

Merged
hedmana merged 3 commits into
mainfrom
feat/recover-citations
Sep 29, 2026
Merged

hedmana merged 3 commits into
mainfrom
feat/recover-citations

Conversation

@hedmana

@hedmana hedmana commented Sep 29, 2026

Copy link
Copy Markdown
Owner

Closes #49

What

  • python -m graph.semanticscholar is an optional stage between crawl and build. For each crawled work whose OpenAlex id is a MAG id, it fetches the Semantic Scholar references, 500 works per request, and backs off on the shared rate limit. Each reference is matched back to a crawled work by MAG id first. Failing that, it matches by exact title within a year, where exactly one work qualifies. Only citations between crawled works are added, and they go to raw/recovered_edges.parquet, kept apart from the crawl's own.
  • graph.build merges recovered citations in when the file exists. Deleting the file and rebuilding undoes them.
  • The API key is read from S2_API_KEY if set, and is never logged. No new dependency.

Why

About one reference in eight that OpenAlex lists points at a record it has since deleted. On the served crawl, the stage asked about 6,811 works and matched 22,582 citations. 6,808 of them were missing from OpenAlex, about 7% more citations. GCN regains Bruna, Henaff, Weston, Duvenaud and GGNN; GraphSAGE regains GCN, Bruna and WL kernels. 2.2% of recovered citations point more than a year forward, the same kind of anomaly cleaning already handles.

gnn.lineages on the draft gold set, before and after:

before after
gold originators reachable 90/94 92/94
learned: gold found in top 10 45 44
learned Recall@10 / nDCG@10 0.484 / 0.466 0.465 / 0.439
learned top-10 overlap, 10% of citations dropped 0.844 0.894

The recovered citations don't find more gold originators. They make the graph more faithful and every ranking steadier when citations go missing. Recall is measured over reachable originators, so it drops as coverage grows. gnn.evaluate is unchanged within spread.

Verify

uv run pytest tests/test_semanticscholar.py tests/test_build.py
uv run python -m graph.semanticscholar --field gnn   # about 3.5 minutes without a key
uv run python -m graph.build --field gnn

Generated with Claude Code

@hedmana
hedmana merged commit 21ba77e into main Sep 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recover the citations OpenAlex lost

1 participant