Repository navigation
feat: crawl forward from the seeds and without a cap - #40
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39
What
crawl.forward_depthin the field config: once the backward crawl is done, the crawler takes in the works citing the seeds, that many hops out.gnn.yamlsets it to 1. It requirestopic_id, which bounds the forward crawl along with the date range, so it cannot wander into every field that ever cited a seed. The forward phase checkpoints per work and resumes like the backward one.OpenAlexClient.citing_workstakes the same topic and year bounds asworks_by_topic, through one shared filter builder.gnn.yamlseeds GAT by its primary OpenAlex recordW2766453196. The old seed was a secondary record with no references, so cleaning dropped it.stats.jsonreports the count asredated_nodes._cut_pointssteps a cut back a year when a single year holds more than a split's share of citations. Otherwise that year swallows the later split: on the expanded crawl, one of the three evaluation cuts had an empty test split, and every metric came out NaN. Cuts on the current snapshot are unchanged.PLAN.mdrecords the results.Why
The graph held only the seeds' past. Training stopped at 2010, no GNN-era paper could be traced, and gold originators such as Henaff and ECFP were missing.
On a fresh crawl with these settings (built into
data/expanded, not served):The four gold originators still unreachable are references OpenAlex holds under dead ids. They return 404, and about 12% of all references do. That includes GCN's references to Bruna, Henaff and Planetoid. Recovering them needs a second source.
gnn.evaluateon the expanded snapshot, paired MRR over the logistic baseline: +0.017 ± 0.013 (14/15 wins) with uniform negatives, and +0.002 ± 0.006 (10/15) with year-matched ones. In the GNN era, citations concentrate on GCN, GAT and GraphSAGE. Popularity nearly matches the model once the year is matched, and recency is worthless.Verify
uv run pytest tests/test_crawler.py tests/test_build.py tests/test_dataset.py tests/test_config.py tests/test_openalex.py uv run python -m graph.crawler --field gnn --data-dir data/expanded # about 3 minutes uv run python -m graph.build --field gnn --data-dir data/expanded uv run python -m gnn.evaluate --field gnn --data-dir data/expandedGenerated with Claude Code