Repository navigation
feat: recover citations OpenAlex lost, from Semantic Scholar - #50
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #49
What
python -m graph.semanticscholaris an optional stage between crawl and build. For each crawled work whose OpenAlex id is a MAG id, it fetches the Semantic Scholar references, 500 works per request, and backs off on the shared rate limit. Each reference is matched back to a crawled work by MAG id first. Failing that, it matches by exact title within a year, where exactly one work qualifies. Only citations between crawled works are added, and they go toraw/recovered_edges.parquet, kept apart from the crawl's own.graph.buildmerges recovered citations in when the file exists. Deleting the file and rebuilding undoes them.S2_API_KEYif set, and is never logged. No new dependency.Why
About one reference in eight that OpenAlex lists points at a record it has since deleted. On the served crawl, the stage asked about 6,811 works and matched 22,582 citations. 6,808 of them were missing from OpenAlex, about 7% more citations. GCN regains Bruna, Henaff, Weston, Duvenaud and GGNN; GraphSAGE regains GCN, Bruna and WL kernels. 2.2% of recovered citations point more than a year forward, the same kind of anomaly cleaning already handles.
gnn.lineageson the draft gold set, before and after:The recovered citations don't find more gold originators. They make the graph more faithful and every ranking steadier when citations go missing. Recall is measured over reachable originators, so it drops as coverage grows.
gnn.evaluateis unchanged within spread.Verify
uv run pytest tests/test_semanticscholar.py tests/test_build.py uv run python -m graph.semanticscholar --field gnn # about 3.5 minutes without a key uv run python -m graph.build --field gnnGenerated with Claude Code