Skip to content

fix: fold duplicate records of one work into one - #62

Merged
hedmana merged 4 commits into
mainfrom
fix/fold-duplicates
Sep 30, 2026
Merged

hedmana merged 4 commits into
mainfrom
fix/fold-duplicates

Conversation

@hedmana

@hedmana hedmana commented Sep 30, 2026

Copy link
Copy Markdown
Owner

Closes #61

What

  • graph.build.clean folds duplicate records into one. It keeps the most cited record, together with every citation and any abstract the kept record lacks. The report counts these as merged_nodes.
  • Versions are detected automatically: one title, one first author's family name, and years at most two apart after re-dating. A title alone is not enough, because Deep learning is both LeCun 2015 and Goodfellow 2016. A given name is too strict, because DiffPool's first author is both Rex and Zhitao Ying.
  • A new field-config key, duplicate_works, names records filed under another work's metadata. In gnn.yaml these are GCN, GraphSAGE, GIN, VGAE, Attention Is All You Need, Adam, BERT, Hamilton et al. 2017, APPNP (under its first author's later name), and a 2020 book chapter that OpenAlex credits with GAT's citations. Each one was identified by what its citers cite on Semantic Scholar.
  • GCN's record W2964015378 moves from exclude_works to duplicate_works. Pitfalls and Neural Combinatorial Optimization get their titles back; NCO's authors are still wrong, since no author correction exists. W4292779060 is excluded, because it has too few citers to identify.
  • The Deep Graph Infomax gold target moves to its kept record.

Why

About 450 crawled works sat in the graph twice, which split each work's citations and could rank a work twice. For example, ChebNet had 1446 + 435 citations inside the graph and DiffPool 415 + 414. Excluding GCN's mis-filed record discarded 2458 citations.

On the rebuilt snapshot gnn-20260930, the graph goes from 10051 works to 9599, and GCN's citations inside the graph go from 2453 to 3779.

dev found dev nDCG@10 test found test nDCG@10
learned, before 23/32 0.695 17/31 0.542
learned, after 23/32 0.690 16/31 0.509

This is a metadata fix, not tuning, and the ranking does not improve. The proxy gate still holds, with 15/15 paired MRR wins over logistic on both samplers (+0.023 uniform, +0.014 year-matched). Adam, no longer split across two records, enters GGSNN's top 5, which is the known problem of tools competing with originators.

Verify

uv run pytest tests/test_build.py
uv run python -m graph.build --field gnn   # merged_nodes 463
uv run python -m gnn.lineages --field gnn

Generated with Claude Code

@hedmana
hedmana merged commit fef7660 into main Sep 30, 2026
4 checks passed
@hedmana
hedmana deleted the fix/fold-duplicates branch September 30, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate OpenAlex records split works in two

1 participant