diff --git a/README.md b/README.md index 4ad1ec9..d41ba0a 100644 --- a/README.md +++ b/README.md @@ -2,56 +2,73 @@ [![Tests](https://img.shields.io/github/actions/workflow/status/IssunDB/kaggle-knowledge-graph/tests.yml?label=tests&style=flat&labelColor=333333&logo=github&logoColor=white)](https://github.com/IssunDB/kaggle-knowledge-graph/actions/workflows/tests.yml) [![Code Coverage](https://img.shields.io/codecov/c/github/IssunDB/kaggle-knowledge-graph?style=flat&label=coverage&labelColor=333333&logo=codecov&logoColor=white)](https://codecov.io/gh/IssunDB/kaggle-knowledge-graph) +[![Hugging Face](https://img.shields.io/badge/HF%20Dataset-view-ffd21e?style=flat&labelColor=333333&logo=huggingface&logoColor=white)](https://huggingface.co/datasets/habedi/kaggle-knowledge-graph) [![Examples](https://img.shields.io/badge/examples-view-green?style=flat&labelColor=282c34&logo=neo4j)](https://github.com/IssunDB/kaggle-knowledge-graph/tree/main/examples) [![License](https://img.shields.io/badge/license-MIT-007ec6?style=flat&labelColor=333333&logo=open-source-initiative&logoColor=white)](https://github.com/IssunDB/kaggle-knowledge-graph/blob/main/LICENSE) --- This repository contains code for building a knowledge graph from the [Meta Kaggle](https://www.kaggle.com/datasets/kaggle/meta-kaggle) dataset -and loading it into [IssunDB](https://github.com/IssunDB/issun-db), which provides CLI and MCP interfaces for querying the data. +and loading it into [IssunDB](https://github.com/IssunDB/issun-db/releases), which provides CLI and MCP interfaces for querying the data. ## Quickstart ### Data -#### 1. Download Dataset +The pipeline needs the Meta Kaggle tabular dataset, and it can (optionally) include the Meta Kaggle Code dataset. + +#### 1. Download Meta Kaggle Dataset ```bash curl -L -o /path/to/meta-kaggle.zip \ https://www.kaggle.com/api/v1/datasets/download/kaggle/meta-kaggle ``` -#### 2. Extract Dataset +#### 2. Extract Meta Kaggle Dataset ```bash unzip /path/to/meta-kaggle.zip -d /path/to/meta-kaggle ``` -#### 3. Configure Environment Variables +#### 3. Download and Extract Meta Kaggle Code (Optional) + +```bash +curl -L -o /path/to/meta-kaggle-code.zip \ + https://www.kaggle.com/api/v1/datasets/download/kaggle/meta-kaggle-code + +unzip /path/to/meta-kaggle-code.zip -d /path/to/meta-kaggle-code +``` + +#### 4. Configure Environment Variables ```bash export META_KAGGLE_DIR="/path/to/meta-kaggle" + +# This is optional. Set only if Meta Kaggle Code was downloaded export META_KAGGLE_CODE_DIR="/path/to/meta-kaggle-code" ``` +> [!NOTE] +> Remember to replace the correct paths in the commands above. + ### Build and Launch Build the Kaggle knowledge graph and launch the CLI or MCP server: -- `make graph-kc` builds the competition-centered knowledge graph from the Meta Kaggle dataset (needs `META_KAGGLE_CODE_DIR` for import and API call - parsing). +- `make graph-kc` builds the knowledge graph from the Meta Kaggle dataset (parses imports and API calls if `META_KAGGLE_CODE_DIR` is set). - `make comp-cli` opens the competition knowledge graph in the IssunDB CLI. - `make comp-mcp` runs the IssunDB MCP server for the competition knowledge graph. -- `make graph-kernel` builds the kernel-centered knowledge graph from the Meta Kaggle dataset (needs `META_KAGGLE_CODE_DIR` for import and API call - parsing). +- `make graph-kernel` builds the knowledge graph from the Meta Kaggle dataset (parses imports and API calls if `META_KAGGLE_CODE_DIR` is set). - `make kernel-cli` opens the kernel knowledge graph in the IssunDB CLI. - `make kernel-mcp` runs the IssunDB MCP server for the kernel knowledge graph. - `make help` shows all available Makefile targets. -### Publish the Graph as a Dataset +### Publish the Graph as a Hugging Face Dataset -- `make hf-package HF_SNAPSHOT=YYYY-MM-DD` packages the staged competition graph as a Hugging Face dataset in `databases/hf-dataset`, with the Parquet - files under `data/`, a dataset card, and a manifest of row counts and checksums. +The published dataset is available on Hugging Face at [habedi/kaggle-knowledge-graph](https://huggingface.co/datasets/habedi/kaggle-knowledge-graph). + +- `make hf-package HF_SNAPSHOT=YYYY-MM-DD` packages the competition graph as a Hugging Face dataset in `databases/hf-dataset`, with the Parquet + files under `data/`, a dataset card, and other metadata. - `make hf-upload HF_REPO_ID=/ HF_VERSION=` uploads that directory to the Hugging Face Hub and tags it with the release version (log in first with `hf auth login`). @@ -75,8 +92,6 @@ To connect AI agents to the IssunDB MCP server, use the configuration template a } ``` -Replace `/path/to/kaggle-knowledge-graph` with the absolute path to your repository root directory. - ### Knowledge Graph Schema
diff --git a/scripts/issundb_load.py b/scripts/issundb_load.py index 1f1803c..bbf638a 100644 --- a/scripts/issundb_load.py +++ b/scripts/issundb_load.py @@ -84,7 +84,9 @@ def validate(log_text: str, expected_nodes: dict[str, int]) -> list[str]: # Any command-level error fails the load (including a rejected constraint). for line in log_text.splitlines(): - low = line.lower() + low = line.lower().strip() + if low.startswith("issundb>"): + low = low[len("issundb>") :].strip() if low.startswith("error") or "mdb_" in low or "storage dependency error" in low: failures.append(f"command error: {line.strip()}") diff --git a/scripts/package_hf_dataset.py b/scripts/package_hf_dataset.py index c170dcb..30f9f5b 100644 --- a/scripts/package_hf_dataset.py +++ b/scripts/package_hf_dataset.py @@ -171,8 +171,6 @@ def _front_matter(tables: list[TableInfo]) -> str: "- knowledge-graph", "- graph", "- competitions", - "- notebooks", - "- code-analysis", "size_categories:", f"- {_size_category(total_rows)}", "configs:", @@ -203,17 +201,13 @@ def render_card(tables: list[TableInfo], meta: ReleaseMeta) -> str: body = f""" # Kaggle Knowledge Graph -A competition-centered knowledge graph built from Kaggle's public -[Meta Kaggle]({META_KAGGLE_URL}) and [Meta Kaggle Code]({META_KAGGLE_CODE_URL}) -datasets. It links competitions, teams, submissions, users, notebooks, notebook -versions, datasets, discussion forums, tags, organizations, and the libraries -and API calls used in notebook code. +A knowledge graph built from Kaggle's public +[Meta Kaggle]({META_KAGGLE_URL}) (and [Meta Kaggle Code]({META_KAGGLE_CODE_URL}) +datasets). It links competitions, teams, submissions, users, notebooks, +datasets, discussion forums, tags, organizations, and notebook code invocations. -The graph is shipped as plain Parquet files, one per node label and one per -relationship type, so it can be loaded with DuckDB, Polars, pandas, Spark, or -any graph database that bulk-imports tabular data. Every node file has an `Id` -column, and every edge file has two columns holding the source and destination -`Id` values. +**See the [project repository]({SOURCE_REPO_URL}) for build scripts and +documentation on the graph schema, data model, and usage examples.** ## Release @@ -221,35 +215,24 @@ def render_card(tables: list[TableInfo], meta: ReleaseMeta) -> str: | --- | --- | | Version | `{meta.version}` | | Meta Kaggle snapshot | {meta.snapshot} | -| Packaged on | {meta.packaged_on} | | Build code | [{SOURCE_REPO_URL}]({SOURCE_REPO_URL}) at commit `{meta.source_commit}` | | Nodes | {total_nodes:,} | | Edges | {total_edges:,} | | Size | {total_bytes / (1 << 20):,.0f} MiB | -The `manifest.json` file beside this card lists the row count, byte size, and -SHA-256 digest of every file in this release. +File counts, byte sizes, and SHA-256 digests are recorded in `manifest.json`. ## Scope -The graph covers competitions enabled on or after 2020-01-01, excluding -Community (in-class) events, and everything the build could attach to them: - -- Ranked or medal-winning teams, their members, and their leaders. -- Each team's selected submissions and its public and private leaderboard - submissions. -- The 50 most voted notebooks per competition, with the notebook versions that - were sourced from that competition, their authors, and their fork lineage. -- Datasets and dataset versions used by those notebook versions. -- The competition discussion forums with every topic and message, including - team write-up topics. Forum message text is included as the original HTML in - `Message` and as Markdown in `RawMarkdown`. -- The tag taxonomy plus the host and dataset-owner organizations. -- Library imports parsed from the notebook source, and Python API calls parsed - with Tree-sitter and resolved to the imported module (for example - `numpy.mean`). - -Every edge file references only nodes that are present in this release. +The graph covers competitions enabled on or after 2020-01-01 (excluding Community events) and their + associated entities: + +- Ranked and medal-winning teams, their members, leaders, and leaderboard submissions. +- The 50 most-voted notebooks per competition, notebook version lineage, and referenced datasets. +- Competition discussion forums with topics, team write-ups, and messages (in raw HTML and Markdown). +- Tag taxonomy, host and owner organizations, imported libraries, and parsed Python API calls. + +All edge endpoints resolve to nodes present in this release. ## Tables @@ -260,7 +243,7 @@ def render_card(tables: list[TableInfo], meta: ReleaseMeta) -> str: """ for table in nodes: body += ( - f"| `{table.label}` | `{table.filename}` | {table.rows} | {_columns_cell(table)} |\n" + f"| `{table.label}` | `{table.filename}` | {table.rows:,} | {_columns_cell(table)} |\n" ) body += """ @@ -272,29 +255,28 @@ def render_card(tables: list[TableInfo], meta: ReleaseMeta) -> str: for table in edges: body += ( f"| `{table.etype}` | `{table.src}` | `{table.dst}` | `{table.filename}` " - f"| {table.rows} |\n" + f"| {table.rows:,} |\n" ) body += f""" ## Usage -Query the files directly with DuckDB: +Query directly with DuckDB: ```python import duckdb -con = duckdb.connect() -con.execute(\"\"\" - SELECT c.Title, count(*) AS teams +duckdb.sql(\"\"\" + SELECT c.Title, COUNT(*) AS teams FROM 'data/edges_team_competed_in_competition.parquet' e JOIN 'data/nodes_competition.parquet' c ON c.Id = e.to_competition_id GROUP BY c.Title ORDER BY teams DESC LIMIT 10 -\"\"\").fetchall() +\"\"\").show() ``` -Or with the `datasets` library, one configuration per table: +Load with Hugging Face `datasets`: ```python from datasets import load_dataset @@ -302,45 +284,23 @@ def render_card(tables: list[TableInfo], meta: ReleaseMeta) -> str: competitions = load_dataset("{meta.repo_id}", "nodes_competition", split="train") ``` -To query the graph with Cypher, load the files into -[IssunDB](https://github.com/habedi/issun-db) using the build repository's -`make comp-load` target, which also starts an MCP server for AI assistants. - -## Building It Yourself - -The build code lives at [{SOURCE_REPO_URL}]({SOURCE_REPO_URL}). It stages the -Meta Kaggle CSVs with DuckDB, parses imports with Polars, parses Python API -calls with Tree-sitter, and validates that no staged edge points to a missing -node. The seed rules are fixed, so the same Meta Kaggle export reproduces the -same graph. +To query with Cypher, load the files into [IssunDB](https://github.com/IssunDB/issun-db) or +any other graph database of your choice. ## Limitations -- Only competitions enabled since 2020 are included, and only the 50 most voted - notebooks per competition. -- The graph is a snapshot of the Meta Kaggle export dated {meta.snapshot}. - Later activity on Kaggle is absent. -- Library and API call tables cover only notebook versions whose source is - present in the Meta Kaggle Code export. -- Forum messages are stored as Kaggle's raw HTML and Markdown and are not - cleaned. - -## Licensing and Attribution - -Meta Kaggle and Meta Kaggle Code are published by Kaggle under the -[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license, -and this derived dataset carries the same license. Use is limited to -non-commercial purposes, and derivatives must be shared under the same terms. -Please credit Kaggle as the source of the underlying data. - -The build code is separately licensed under the MIT License. +- Snapshot from {meta.snapshot}; later Kaggle activity is not included. +- Covers competitions enabled on or after 2020-01-01 and at most 50 most-voted notebooks per competition. +- Library and API call tables cover only notebook versions available in Meta Kaggle Code. +- Forum messages are stored as raw Kaggle HTML and Markdown. -## Personal Data +## Licensing and Privacy -The graph contains public Kaggle profile fields (user names, display names, -countries, and performance tiers) and public forum posts, exactly as published -in Meta Kaggle. Nothing is added beyond what Kaggle already publishes. If you -find content that should not be here, open an issue on the build repository. +Meta Kaggle data is licensed under +[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) for +non-commercial use with attribution. +Build scripts at [{SOURCE_REPO_URL}]({SOURCE_REPO_URL}) are licensed under the MIT License. +The dataset includes only public profile attributes and forum posts published by Kaggle. """ return _front_matter(tables) + "\n" + body