Skip to content

feat: crawl forward from the seeds and without a cap - #40

Merged
hedmana merged 6 commits into
mainfrom
feat/crawl-expansion
Sep 29, 2026
Merged

hedmana merged 6 commits into
mainfrom
feat/crawl-expansion

Conversation

@hedmana

@hedmana hedmana commented Sep 29, 2026

Copy link
Copy Markdown
Owner

Closes #39

What

  • crawl.forward_depth in the field config: once the backward crawl is done, the crawler takes in the works citing the seeds, that many hops out. gnn.yaml sets it to 1. It requires topic_id, which bounds the forward crawl along with the date range, so it cannot wander into every field that ever cited a seed. The forward phase checkpoints per work and resumes like the backward one.
  • OpenAlexClient.citing_works takes the same topic and year bounds as works_by_topic, through one shared filter builder.
  • gnn.yaml seeds GAT by its primary OpenAlex record W2766453196. The old seed was a secondary record with no references, so cleaning dropped it.
  • Cleaning re-dates works that OpenAlex dates more than a year after the earliest work citing them. They take that citer's year. Examples are a 1990 paper dated 2018, "Attention Is All You Need" dated 2025, and a mis-merged record cited 2,347 times dated 2025. Without this, every citation of such a work points into the future, which the temporal split reads as a free label. stats.json reports the count as redated_nodes.
  • _cut_points steps a cut back a year when a single year holds more than a split's share of citations. Otherwise that year swallows the later split: on the expanded crawl, one of the three evaluation cuts had an empty test split, and every metric came out NaN. Cuts on the current snapshot are unchanged.
  • PLAN.md records the results.

Why

The graph held only the seeds' past. Training stopped at 2010, no GNN-era paper could be traced, and gold originators such as Henaff and ECFP were missing.

On a fresh crawl with these settings (built into data/expanded, not served):

before after
works 2,789 9,249
citations 15,155 103,539
test windows after 2008, 2010, 2013 after 2022, 2023, 2024
gold originators reachable 88/94 90/94

The four gold originators still unreachable are references OpenAlex holds under dead ids. They return 404, and about 12% of all references do. That includes GCN's references to Bruna, Henaff and Planetoid. Recovering them needs a second source.

gnn.evaluate on the expanded snapshot, paired MRR over the logistic baseline: +0.017 ± 0.013 (14/15 wins) with uniform negatives, and +0.002 ± 0.006 (10/15) with year-matched ones. In the GNN era, citations concentrate on GCN, GAT and GraphSAGE. Popularity nearly matches the model once the year is matched, and recency is worthless.

Verify

uv run pytest tests/test_crawler.py tests/test_build.py tests/test_dataset.py tests/test_config.py tests/test_openalex.py
uv run python -m graph.crawler --field gnn --data-dir data/expanded   # about 3 minutes
uv run python -m graph.build --field gnn --data-dir data/expanded
uv run python -m gnn.evaluate --field gnn --data-dir data/expanded

Generated with Claude Code

@hedmana
hedmana merged commit e9c1953 into main Sep 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crawl the field forward and without a cap

1 participant