diff --git a/README.md b/README.md index 2a934ea..1822ff8 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,8 @@ description: > topic_id: null # OpenAlex topic delimiting the field seed_papers: [] # OpenAlex work IDs the crawl starts from crawl: - hop_depth: 2 + hop_depth: 2 # references, backwards from the seeds + forward_depth: 1 # works citing the seeds, within topic_id date_range: { start: null, end: null } ``` diff --git a/fields/gnn.yaml b/fields/gnn.yaml index ecc3f5b..b995ff6 100644 --- a/fields/gnn.yaml +++ b/fields/gnn.yaml @@ -29,11 +29,15 @@ seed_papers: - W2519887557 # Semi-Supervised Classification with Graph Convolutional Networks (2016) - W2606780347 # Neural Message Passing for Quantum Chemistry (2017) - W2624431344 # Inductive Representation Learning on Large Graphs (2017) - - W4297733535 # Graph Attention Networks (2017) + - W2766453196 # Graph Attention Networks (2017) crawl: # How many citation hops to expand outward from the seeds. hop_depth: 2 + # How many hops forward, to the works citing the seeds. Only works under + # topic_id are taken in, or the crawl would follow every field that ever + # cited a seed. + forward_depth: 1 # Publication years to keep. Bounds are inclusive; null means unbounded. date_range: start: 1980 diff --git a/src/gnn/PLAN.md b/src/gnn/PLAN.md index 5e5a5a6..565c4a8 100644 --- a/src/gnn/PLAN.md +++ b/src/gnn/PLAN.md @@ -18,12 +18,24 @@ Written 2026-09-29 against snapshot `gnn-20260924`. `logistic` is a logistic regression on year gap and target in-degree. Paired MRR over it, per run: uniform +0.010 ± 0.013 (11/15 wins), year-matched +0.031 ± 0.013 (15/15). The proxy gate holds under year-matched negatives; under uniform only AUC clears it (15/15), MRR is within spread. +On the expanded snapshot (`gnn-20260929`, 9249 works, tested after 2022, 2023, 2024): + +| Sampler | Scorer | AUC | MRR | Hits@10 | +|---|---|---|---|---| +| uniform | model | 0.926 ± 0.014 | 0.523 ± 0.012 | 0.783 ± 0.018 | +| uniform | recency | 0.527 ± 0.003 | 0.029 ± 0.002 | 0.029 ± 0.012 | +| uniform | logistic | 0.918 ± 0.004 | 0.506 ± 0.012 | 0.757 ± 0.016 | +| year_matched | model | 0.879 ± 0.016 | 0.447 ± 0.014 | 0.729 ± 0.009 | +| year_matched | logistic | 0.857 ± 0.005 | 0.444 ± 0.009 | 0.725 ± 0.005 | + +Paired MRR over logistic: uniform +0.017 ± 0.013 (14/15), year-matched +0.002 ± 0.006 (10/15). In the GNN era citations pile onto a few hubs, so popularity alone nearly matches the model once the year is matched, and recency is worthless. + Why the model lost before: - Val/test sources have no out-edges in their message graph (cold start); every train positive was a message edge. Best epoch 3 of 23. Fixed by cold-start training. - Negatives were sampled once; degree features included supervision edges. Fixed by cold-start training. - Uniform negatives make recency sufficient. - `learned_flow` weights a paper's existing references; nothing measures that. -- Crawl is backward-only, capped at 3000 works, 81% of edges dangle. Train cut is 2010, so it mostly learns pre-GNN citation habits. +- Crawl was backward-only and capped at 3000 works, so training cut at 2010 and learned pre-GNN citation habits. Fixed by crawl expansion, except references OpenAlex holds under dead ids. ## Gates @@ -41,16 +53,17 @@ One PR each, named for what it does. - Hashed title words: -0.02, overfits. - `topic_id`: +0.06, but it leaks. OpenAlex assigns topics from citations as of the crawl, the reason `cited_by_count` is excluded. - Held-out share, learning rate, depth, width, dropout, weight decay, and MRR instead of AP for checkpoint selection: nothing beyond noise. -- [ ] **Originator ranking evaluation.** Gold lineages for 20-30 targets in `fields/gnn.gold.yaml`, from survey history sections. nDCG@10 and Recall@10 vs all baselines. Ablation: `learned_flow` with uniform edge weights. Stability: top-10 overlap across seeds and 10% edge dropout. +- [ ] **Originator ranking evaluation.** Gold lineages for 20-30 targets in `fields/gold/gnn.yaml` (not `fields/*.yaml`, which the API lists as fields), from survey history sections. nDCG@10 and Recall@10 vs all baselines. Ablation: `learned_flow` with uniform edge weights. Stability: top-10 overlap across seeds and 10% edge dropout. - [ ] **Citation influence labels.** Semantic Scholar `isInfluential` per citation; per-paper AUC of the decoder on influential vs incidental references. - [ ] **Influence-aligned objective.** Fine-tune the decoder listwise on influential references, link prediction as pretraining. Validate on the gold set. -- [ ] **Crawl expansion** (parallel, after link evaluation). Raise `max_nodes`, add forward expansion via `citing_works` filtered by `topic_id`. Rerun link evaluation. +- [x] **Crawl expansion.** Uncapped backward crawl, plus `forward_depth: 1`: the in-topic works citing the seeds. 2789 works and 15k citations became 9249 and 104k. GAT is seeded by its primary record. Cleaning re-dates 262 works OpenAlex dates years after their citers, and a year heavier than its split's share no longer empties the split after it. Gold originators reachable from their targets: 88/94 to 90/94. The other four are references OpenAlex holds under dead ids (about 12% of all references), among them GCN's to Bruna, Henaff and Planetoid. Optional: blinded LLM pairwise judge, trusted only after it agrees with the gold set. ## Open decisions -- Uniform MRR is within spread: accept the proxy gate on year-matched and AUC, or do crawl expansion before originator ranking evaluation. -- Semantic Scholar as a second data source (citation influence labels). +- Serve the expanded snapshot: move `data/expanded/gnn/snapshots/gnn-20260929` into `data/gnn/snapshots`, then `gnn.dataset` and `gnn.train`. +- On the expanded snapshot the model only ties popularity under year-matched negatives: revisit training there, or let originator ranking evaluation decide. +- Semantic Scholar as a second data source: citation influence labels, and the references OpenAlex holds under dead ids. - Who verifies the gold set (originator ranking evaluation). - Abstract embeddings need a new dependency; deferred until originator ranking evaluation shows text is the bottleneck. diff --git a/src/gnn/dataset.py b/src/gnn/dataset.py index 3686c71..15acfd3 100644 --- a/src/gnn/dataset.py +++ b/src/gnn/dataset.py @@ -148,12 +148,21 @@ def publication_order(x: Tensor) -> Tensor: def _cut_points(edge_years: Tensor, val_fraction: float, test_fraction: float) -> tuple[int, int]: - """The last training year and the last validation year, by share of citations.""" + """The last training year and the last validation year, by share of citations. + + A year's citations cannot be divided, so a year holding more than a split's + share would swallow the split after it. The cut steps back a year instead. + """ ordered = edge_years.sort().values count = ordered.numel() - train_end = max(int(count * (1 - val_fraction - test_fraction)) - 1, 0) - val_end = max(int(count * (1 - test_fraction)) - 1, 0) - return int(ordered[train_end]), int(ordered[val_end]) + train_until = int(ordered[max(int(count * (1 - val_fraction - test_fraction)) - 1, 0)]) + val_until = int(ordered[max(int(count * (1 - test_fraction)) - 1, 0)]) + years = ordered.unique() + if test_fraction and val_until == years[-1] and years.numel() > 1: + val_until = int(years[-2]) + if val_fraction and train_until >= val_until and (earlier := years[years < val_until]).numel(): + train_until = int(earlier[-1]) + return train_until, val_until def sample_negatives( diff --git a/src/graph/build.py b/src/graph/build.py index 781c0ee..1922458 100644 --- a/src/graph/build.py +++ b/src/graph/build.py @@ -24,6 +24,9 @@ logger = logging.getLogger(__name__) +# A preprint can be cited a year before the venue date OpenAlex gives it. +PREPRINT_SLACK = 1 + NODE_ATTRIBUTES = ( "title", "abstract", @@ -45,6 +48,7 @@ class CleaningReport: duplicate_edges: int = 0 out_of_range_nodes: int = 0 dangling_edges: int = 0 + redated_nodes: int = 0 self_loops: int = 0 disconnected_nodes: int = 0 nodes: int = 0 @@ -124,6 +128,7 @@ def clean( connected = edges["source"].isin(known) & edges["target"].isin(known) report.dangling_edges = int((~connected).sum()) edges = edges[connected] + nodes, report.redated_nodes = _redate(nodes, edges) if largest_component_only: keep = _largest_component(nodes["id"], edges) @@ -136,6 +141,24 @@ def clean( return nodes, edges, report +def _redate(nodes: pd.DataFrame, edges: pd.DataFrame) -> tuple[pd.DataFrame, int]: + """Pull back any work dated well after the works that cite it. + + OpenAlex sometimes dates a work by a late reprint or a mis-merged record. A + work cited in 2017 but dated 2025 turns every citation of it into one of the + future, which a temporal split reads as a free label. Such a work takes the + year of its earliest citer. + """ + if edges.empty: + return nodes, 0 + year = nodes.set_index("id")["publication_year"] + earliest = nodes["id"].map(edges["source"].map(year).groupby(edges["target"]).min()) + late = nodes["publication_year"] > earliest + PREPRINT_SLACK + nodes = nodes.copy() + nodes.loc[late, "publication_year"] = earliest[late].astype(nodes["publication_year"].dtype) + return nodes, int(late.sum()) + + def _largest_component(node_ids: pd.Series, edges: pd.DataFrame) -> set[str]: scaffold = nx.Graph() scaffold.add_nodes_from(node_ids) diff --git a/src/graph/config.py b/src/graph/config.py index 68f698f..89219fe 100644 --- a/src/graph/config.py +++ b/src/graph/config.py @@ -5,7 +5,7 @@ from pathlib import Path import yaml -from pydantic import BaseModel, Field, field_validator +from pydantic import BaseModel, Field, field_validator, model_validator FIELDS_DIR = Path(__file__).resolve().parents[2] / "fields" @@ -22,6 +22,7 @@ def contains(self, year: int | None) -> bool: class CrawlConfig(BaseModel): hop_depth: int = Field(ge=0) + forward_depth: int = Field(default=0, ge=0) date_range: DateRange = Field(default_factory=DateRange) @@ -38,6 +39,14 @@ class FieldConfig(BaseModel): def _collapse_whitespace(cls, value: str) -> str: return " ".join(value.split()) + @model_validator(mode="after") + def _bound_the_forward_crawl(self) -> FieldConfig: + # A seed's citers span every field that ever borrowed from it; the topic + # is the only thing keeping a forward crawl inside this one. + if self.crawl.forward_depth and not self.topic_id: + raise ValueError("crawl.forward_depth needs a topic_id to bound it") + return self + def load_field(name: str, fields_dir: Path | None = None) -> FieldConfig: """Load `fields/.yaml`, or a path to a config file.""" diff --git a/src/graph/crawler.py b/src/graph/crawler.py index b276893..eabdde4 100644 --- a/src/graph/crawler.py +++ b/src/graph/crawler.py @@ -31,6 +31,7 @@ class CrawlState: depth: int = 0 frontier: list[str] = field(default_factory=list) next_frontier: list[str] = field(default_factory=list) + forward: bool = False class CrawlJournal: @@ -99,6 +100,11 @@ class SnowballCrawler: lineage runs that way. Works outside the field's date range are still recorded, but are not expanded through: they bound the crawl without silently dropping edges the cleaning stage may want to see. + + Backwards alone, the graph is the seeds' past and nothing after them: the + work the field went on to do is neither traceable nor there to learn from. + So once the backward crawl is done, a forward one takes in the works citing + the seeds, `forward_depth` hops out, kept inside the field by its topic. """ def __init__( @@ -122,7 +128,7 @@ def run(self) -> CrawlState: frontier=_dedup(self.config.seed_papers, set()) ) - while state.depth <= self.config.crawl.hop_depth: + while not state.forward and state.depth <= self.config.crawl.hop_depth: pending = _dedup(state.frontier, visited) logger.info("depth %d: %d works to fetch", state.depth, len(pending)) expand = state.depth < self.config.crawl.hop_depth @@ -145,8 +151,44 @@ def run(self) -> CrawlState: if not state.frontier: break + if not state.forward: + state = CrawlState(frontier=list(self.config.seed_papers), forward=True) + self.journal.save_state(state) + + while state.depth < self.config.crawl.forward_depth and state.frontier: + logger.info("forward %d: %d works to expand", state.depth + 1, len(state.frontier)) + while state.frontier: + citing = list(self._citing(state.frontier[0])) + fresh = _dedup((work.id for work in citing), visited) + if self.max_nodes: + fresh = fresh[: max(self.max_nodes - len(visited), 0)] + chosen = set(fresh) + self.journal.append(work for work in citing if work.id in chosen) + visited.update(fresh) + if self.max_nodes and len(visited) >= self.max_nodes: + # The work stays on the frontier, so a resumed crawl asks again. + logger.info("stopping at max_nodes=%d", self.max_nodes) + return state + state.next_frontier.extend(work.id for work in citing) + state.frontier = state.frontier[1:] + self.journal.save_state(state) + + state.depth += 1 + state.frontier = _dedup(state.next_frontier, set()) + state.next_frontier = [] + self.journal.save_state(state) + return state + def _citing(self, work_id: str) -> Iterator[Work]: + date_range = self.config.crawl.date_range + return self.client.citing_works( + work_id, + topic_id=self.config.topic_id, + from_year=date_range.start, + to_year=date_range.end, + ) + def _fetch( self, pending: list[str], diff --git a/src/graph/openalex.py b/src/graph/openalex.py index c7ad4ad..c7f2cd3 100644 --- a/src/graph/openalex.py +++ b/src/graph/openalex.py @@ -83,6 +83,23 @@ def _flatten(cls, payload: Any) -> Any: } +def _filter( + *conditions: str, + topic_id: str | None = None, + from_year: int | None = None, + to_year: int | None = None, +) -> str: + """Join filter conditions, adding a topic and publication-year bounds where given.""" + filters = list(conditions) + if topic_id is not None: + filters.append(f"primary_topic.id:{strip_id_prefix(topic_id)}") + if from_year is not None: + filters.append(f"from_publication_date:{from_year}-01-01") + if to_year is not None: + filters.append(f"to_publication_date:{to_year}-12-31") + return ",".join(filters) + + def _chunked(values: Iterable[str], size: int) -> Iterator[list[str]]: chunk: list[str] = [] for value in values: @@ -156,16 +173,25 @@ def works_by_topic( to_year: int | None = None, ) -> Iterator[Work]: """Enumerate works under a topic, optionally bounded by publication year.""" - filters = [f"primary_topic.id:{strip_id_prefix(topic_id)}"] - if from_year is not None: - filters.append(f"from_publication_date:{from_year}-01-01") - if to_year is not None: - filters.append(f"to_publication_date:{to_year}-12-31") - yield from self._paginate({"filter": ",".join(filters)}) - - def citing_works(self, work_id: str) -> Iterator[Work]: - """Works that cite the given work, i.e. its descendants.""" - yield from self._paginate({"filter": f"cites:{strip_id_prefix(work_id)}"}) + filter_ = _filter(topic_id=topic_id, from_year=from_year, to_year=to_year) + yield from self._paginate({"filter": filter_}) + + def citing_works( + self, + work_id: str, + *, + topic_id: str | None = None, + from_year: int | None = None, + to_year: int | None = None, + ) -> Iterator[Work]: + """Works that cite the given work, i.e. its descendants, optionally within a topic.""" + filter_ = _filter( + f"cites:{strip_id_prefix(work_id)}", + topic_id=topic_id, + from_year=from_year, + to_year=to_year, + ) + yield from self._paginate({"filter": filter_}) def referenced_works(self, work_id: str) -> Iterator[Work]: """Works the given work cites, i.e. its immediate ancestors.""" diff --git a/tests/test_build.py b/tests/test_build.py index 35cd9ba..0855e18 100644 --- a/tests/test_build.py +++ b/tests/test_build.py @@ -75,6 +75,27 @@ def test_nodes_outside_the_date_range_are_dropped_with_their_edges(): assert list(edges.itertuples(index=False, name=None)) == [("A", "C")] +def test_a_work_dated_after_its_citers_takes_its_earliest_citers_year(): + nodes, edges = frames( + [node("A", year=2017), node("B", year=2019), node("C", year=2025)], + [("A", "C"), ("B", "C")], + ) + + nodes, _, report = clean(nodes, edges) + + assert report.redated_nodes == 1 + assert nodes.set_index("id").loc["C", "publication_year"] == 2017 + + +def test_a_preprint_cited_a_year_before_its_venue_date_keeps_it(): + nodes, edges = frames([node("A", year=2016), node("B", year=2017)], [("A", "B")]) + + nodes, _, report = clean(nodes, edges) + + assert report.redated_nodes == 0 + assert nodes.set_index("id").loc["B", "publication_year"] == 2017 + + def test_only_the_largest_component_survives_by_default(): nodes, edges = frames( [node("A"), node("B"), node("C"), node("X"), node("Y")], diff --git a/tests/test_config.py b/tests/test_config.py index 970ccaf..0aa6350 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -58,6 +58,18 @@ def test_name_must_match_filename(tmp_path): load_field("onfile", tmp_path) +def test_a_forward_crawl_needs_a_topic_to_bound_it(tmp_path): + path = write_config(tmp_path) + path.write_text( + path.read_text() + .replace("topic_id: T1", "topic_id: null") + .replace("hop_depth: 2", "hop_depth: 2\n forward_depth: 1") + ) + + with pytest.raises(ValidationError, match="forward_depth needs a topic_id"): + load_field("test", tmp_path) + + def test_negative_hop_depth_is_rejected(tmp_path): write_config(tmp_path, hop_depth=-1) diff --git a/tests/test_crawler.py b/tests/test_crawler.py index ad3e654..1c35369 100644 --- a/tests/test_crawler.py +++ b/tests/test_crawler.py @@ -24,6 +24,7 @@ def __init__(self, citations=None, years=None): self.citations = citations or CITATIONS self.years = years or {} self.requested = [] + self.cited = [] def works(self, work_ids): work_ids = list(work_ids) @@ -37,15 +38,28 @@ def works(self, work_ids): referenced_works=self.citations[work_id], ) + def citing_works(self, work_id, **bounds): + self.cited.append((work_id, bounds)) + citing = [work for work, references in self.citations.items() if work_id in references] + yield from self.works(citing) -def make_config(hop_depth=2, start=None, end=None, seeds=("A",)): + +# The works citing A, and one citing those in turn, which only a backward crawl misses. +DESCENDANTS = {**CITATIONS, "X": ["A", "B"], "Y": ["A", "Z"], "W": ["X"]} + + +def make_config(hop_depth=2, start=None, end=None, seeds=("A",), forward_depth=0): return FieldConfig( name="test", display_name="Test", description="test field", topic_id="T1", seed_papers=list(seeds), - crawl=CrawlConfig(hop_depth=hop_depth, date_range=DateRange(start=start, end=end)), + crawl=CrawlConfig( + hop_depth=hop_depth, + forward_depth=forward_depth, + date_range=DateRange(start=start, end=end), + ), ) @@ -163,6 +177,47 @@ def test_empty_journal_produces_empty_tables(tmp_path): assert pd.read_parquet(edges_path).empty +def test_a_forward_crawl_takes_in_the_works_citing_the_seeds(tmp_path): + client = FakeClient(DESCENDANTS) + journal, _ = crawl(tmp_path, make_config(forward_depth=1), client) + + assert node_ids(journal) == ["A", "B", "C", "D", "E", "X", "Y"] + assert {("X", "A"), ("X", "B"), ("Y", "A"), ("Y", "Z")} <= set(edge_pairs(journal)) + + +def test_each_forward_hop_reaches_one_generation_further(tmp_path): + journal, _ = crawl(tmp_path, make_config(forward_depth=2), FakeClient(DESCENDANTS)) + + assert "W" in node_ids(journal) + + +def test_the_forward_crawl_stays_inside_the_topic_and_years(tmp_path): + client = FakeClient(DESCENDANTS) + crawl(tmp_path, make_config(forward_depth=1, start=2000, end=2020), client) + + assert client.cited == [("A", {"topic_id": "T1", "from_year": 2000, "to_year": 2020})] + + +def test_without_a_forward_depth_no_citing_works_are_asked_for(tmp_path): + client = FakeClient(DESCENDANTS) + journal, _ = crawl(tmp_path, make_config(), client) + + assert client.cited == [] + assert "X" not in node_ids(journal) + + +def test_an_interrupted_forward_crawl_resumes_without_losing_citers(tmp_path): + config = make_config(forward_depth=1) + crawl(tmp_path, config, FakeClient(DESCENDANTS), max_nodes=6, checkpoint_every=1) + + resumed = FakeClient(DESCENDANTS) + SnowballCrawler(config, resumed, CrawlJournal(tmp_path), checkpoint_every=1).run() + journal = CrawlJournal(tmp_path) + + assert node_ids(journal) == ["A", "B", "C", "D", "E", "X", "Y"] + assert resumed.requested == [["X", "Y"]] + + def test_max_nodes_is_exact_regardless_of_checkpoint_size(tmp_path): journal, _ = crawl(tmp_path, make_config(hop_depth=2), max_nodes=2, checkpoint_every=500) diff --git a/tests/test_dataset.py b/tests/test_dataset.py index 1da016d..57cc097 100644 --- a/tests/test_dataset.py +++ b/tests/test_dataset.py @@ -63,6 +63,16 @@ def test_splits_are_cuts_in_time_not_random_samples(): assert dataset.report.train_edges > dataset.report.val_edges +def test_a_year_heavier_than_its_share_does_not_swallow_the_next_split(): + years = {"W0": 2001, "W1": 2002, "W2": 2003, "W3": 2004} | {f"N{i}": 2005 for i in range(6)} + edges = [("W1", "W0"), ("W2", "W1"), ("W3", "W2")] + [(f"N{i}", "W3") for i in range(6)] + + report = build_dataset(citation_graph(years, edges), val_fraction=0.2, test_fraction=0.2).report + + assert (report.train_until, report.val_until) == (2003, 2004) + assert (report.train_edges, report.val_edges, report.test_edges) == (2, 1, 6) + + def test_evaluation_splits_see_only_the_edges_that_preceded_them(): dataset = build_dataset(cohorts(), val_fraction=0.2, test_fraction=0.2) diff --git a/tests/test_openalex.py b/tests/test_openalex.py index ecc71ff..eb1fdc5 100644 --- a/tests/test_openalex.py +++ b/tests/test_openalex.py @@ -131,6 +131,21 @@ def handler(request): assert captured["filter"] == "cites:W1" +def test_citing_works_can_be_bounded_by_topic_and_years(): + captured = {} + + def handler(request): + captured["filter"] = request.url.params["filter"] + return httpx.Response(200, json={"results": [], "meta": {}}) + + list(client_for(handler).citing_works("W1", topic_id="T11273", from_year=2015, to_year=2020)) + + assert captured["filter"] == ( + "cites:W1,primary_topic.id:T11273," + "from_publication_date:2015-01-01,to_publication_date:2020-12-31" + ) + + def test_works_batches_ids_into_or_filters(): ids = [f"W{n}" for n in range(MAX_IDS_PER_FILTER + 10)] batches = []