Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# .dockerignore for building the self-contained Docker Hub image
# `ortecha/graphrag-toolkit-rdf` (see
# examples/lexical-graph-local-dev/docker/jupyter/Dockerfile.hub).
#
# The build context is the repo ROOT, but only three paths are COPYd into the
# image: lexical-graph/, lexical-graph-contrib/sparql/, and the example
# notebooks. Everything else here just keeps the context small and — for the
# notebooks — keeps secrets and heavy generated data OUT of a published image.
#
# NOTE: the dev stack (start-containers.sh --dev) builds from the ./jupyter
# context, so it is unaffected by this file.

# --- VCS, CI, tooling, OS cruft ---
.git
.github
.gitignore
.gitallowed
.pre-commit-config.yaml
**/.DS_Store
**/.ipynb_checkpoints
**/__pycache__
**/*.pyc
**/.pytest_cache
**/htmlcov
**/.coverage
**/*.egg-info

# --- Local virtualenvs ---
.venv
.venv-*
**/.venv

# --- Repo areas this image does not use ---
graphrag-ui
inputs
byokg-rag
benchmark-tests
integration-tests
docs-site
images
examples/lexical-graph-hybrid-dev
examples/lexical-graph-local-dev/docker

# --- Tests (not needed at runtime) ---
lexical-graph/tests
lexical-graph-contrib/sparql/tests
lexical-graph-contrib/falkordb

# --- Secrets + heavy/generated notebook data (do NOT bake into a published image) ---
examples/lexical-graph-local-dev/notebooks/.env
examples/lexical-graph-local-dev/notebooks/nltk_data
examples/lexical-graph-local-dev/notebooks/artifacts
examples/lexical-graph-local-dev/notebooks/soup
examples/lexical-graph-local-dev/notebooks/extracted
1 change: 1 addition & 0 deletions docs-site/astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ export default defineConfig({
{ label: 'Neptune Database', slug: 'lexical-graph/graph-store-neptune-db' },
{ label: 'Neo4j', slug: 'lexical-graph/graph-store-neo4j' },
{ label: 'FalkorDB', slug: 'lexical-graph/graph-store-falkor-db' },
{ label: 'RDF / SPARQL Stores', slug: 'lexical-graph/graph-store-sparql' },
],
},
{
Expand Down
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import { Card, CardGrid, Code } from '@astrojs/starlight/components';
Bring your own knowledge graph. Plug an existing graph into a multi-strategy KGQA pipeline without re-extracting anything.
</Card>
<Card title="Pluggable storage" icon="document">
Graph stores: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB. Vector stores: Neptune, OpenSearch, Postgres, S3 Vectors.
Graph stores: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB, RDF/SPARQL stores. Vector stores: Neptune, OpenSearch, Postgres, S3 Vectors.
</Card>
<Card title="Open source" icon="github">
Apache 2.0, developed in the open by AWS Labs.
Expand Down
152 changes: 152 additions & 0 deletions docs-site/src/content/docs/lexical-graph/graph-store-sparql.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
---
title: RDF / SPARQL stores
---

The RDF / SPARQL contributor package stores the lexical graph in an existing
SPARQL 1.1 query/update endpoint. Generic standards-compatible endpoints are the
default; Amazon Neptune IAM authentication is an optional transport.

The adapter executes native SPARQL. Builders and retrievers provide a
backend-neutral operation identifier and structured parameters. Property-graph
stores execute their native queries, while the RDF store selects the
corresponding native SPARQL operation.

### Install and register

Install the contributor package from a repository checkout:

```bash
pip install ./lexical-graph-contrib/sparql
```

Register its factory before creating the graph store:

```python
from graphrag_toolkit.lexical_graph.storage import GraphStoreFactory
from graphrag_toolkit_contrib.lexical_graph.storage.graph.sparql import (
SPARQLGraphStoreFactory,
)

GraphStoreFactory.register(SPARQLGraphStoreFactory)
```

Connect the graph store to an existing repository or dataset.

### Generic endpoint

Provide the query URL and, if different, the update URL:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
update_endpoint='https://rdf.example.com/update',
)
```

Supported schemes are:

| Scheme | Endpoint transport |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all these bespoke scheme extensions really nessecary? It measn that developers will have to do some string concatination any time they want to pass in a SPARQL endpoint. I would prefer we found another way to specify specific headers for authentication.

|---|---|
| `sparql://host/path` | HTTP |
| `sparql+http://host/path` | HTTP |
| `sparql+s://host/path` | HTTPS |
| `sparql+https://host/path` | HTTPS |
| `sparql+neptune://host:8182` | HTTPS with Neptune IAM SigV4 |

If `update_endpoint` is omitted, the query URL is used for both operations.

HTTP Basic credentials can be passed in the URL, through `username` and
`password`, or through the `SPARQL_USER` and `SPARQL_PASSWORD` environment
variables. Custom headers support bearer tokens and endpoint-specific headers:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
headers={'Authorization': 'Bearer token'},
)
```

### Amazon Neptune IAM

Install the optional botocore dependency:

```bash
pip install './lexical-graph-contrib/sparql[neptune]'
```

Then use the Neptune scheme and AWS region:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+neptune://my-cluster.cluster-abcdefghijkl.eu-central-1.neptune.amazonaws.com:8182',
region_name='eu-central-1',
)
```

Each request is signed for the `neptune-db` service using credentials from
botocore's standard provider chain. Credentials are resolved on every request,
allowing botocore to refresh temporary role, web-identity, IAM Identity Center,
ECS, and EC2 credentials. IAM transport requires HTTPS.

The same transport is available as a plain RDFLib graph:

```python
from graphrag_toolkit_contrib.lexical_graph.storage.graph.sparql.neptune_iam import (
neptune_iam_graph,
)

graph = neptune_iam_graph(
'https://my-cluster.cluster-abcdefghijkl.eu-central-1.neptune.amazonaws.com:8182',
region_name='eu-central-1',
)
try:
rows = graph.query('SELECT * WHERE { ?s ?p ?o } LIMIT 10')
finally:
graph.close()
```

### Namespaces and tenants

The default schema namespace is
`https://awslabs.github.io/graphrag-toolkit/lexical#`; the default instance
namespace is `https://awslabs.github.io/graphrag-toolkit/lexical/`.

Customize them only when creating a new dataset:

```python
graph_store = GraphStoreFactory.for_graph_store(
'sparql+https://rdf.example.com/query',
update_endpoint='https://rdf.example.com/update',
lexical_prefix='gt',
lexical_schema_namespace='https://example.com/graph/schema#',
lexical_instance_namespace='https://example.com/graph/data/',
)
```

Every tenant, including the default tenant, uses a deterministic named graph.
Writes target that graph explicitly, and reads select it with the standard
SPARQL Protocol `default-graph-uri` parameter.

### RDF model

Sources, chunks, topics, statements, facts, and entities retain their existing
lexical-graph identities. A node becomes a deterministic IRI with an RDF class
and `lg:id`; node properties become literal-valued predicates. Simple edges
become RDF predicates.

Extracted facts record their subject, predicate, and object directly:

```turtle
<fact/f1> a lg:Fact ;
lg:subject <entity/amazon> ;
lg:predicate <relation/produces> ;
lg:object <entity/ec2> ;
lg:supports <statement/s1> ;
lg:value "Amazon PRODUCES EC2" .

<relation/produces> a lg:Relation ;
lg:value "PRODUCES" .
```

This is the toolkit's fact model, with the fact resource carrying its subject,
predicate, object, and supporting statement.
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/lexical-graph/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The graphrag-toolkit [lexical-graph](https://github.com/awslabs/graphrag-toolkit
Source → chunk → topic → statement → fact → entity, all linked. Retrieval can hop between any of these levels.
</Card>
<Card title="Pluggable storage" icon="document">
Graph: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB. Vectors: Neptune, OpenSearch, Postgres, S3 Vectors.
Graph: Amazon Neptune (DB and Analytics), Neo4j, FalkorDB, RDF/SPARQL stores. Vectors: Neptune, OpenSearch, Postgres, S3 Vectors.
</Card>
<Card title="Two-stage indexing" icon="seti:config">
Extract and build run as separate micro-batched pipelines so ingest is continuous and resumable.
Expand Down
3 changes: 2 additions & 1 deletion docs-site/src/content/docs/lexical-graph/storage-model.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Graph stores and vector stores provide connectivity to an *existing* storage ins

### Graph store

Graph stores must support the [openCypher](https://opencypher.org/) property graph query language. Graph construction queries typically use an `UNWIND ... MERGE` idiom to create or update the graph for a [batch of inputs](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/best-practices-content.html#best-practices-content-14). The Neptune graph store implementations override the `GraphStore.node_id()` method to ensure that node ids in the code (e.g. `chunkId`) are mapped to Neptune's `~id` reserved property. Alternative graph store implementations can leave the base implementation of `node_id()` as-is. This will result in node ids being mapped to a property of the same name (i.e. a reference to `chunkId` in the code will be mapped to a `chunkId` property of a node).
The built-in property-graph stores use [openCypher](https://opencypher.org/). Graph construction queries typically use an `UNWIND ... MERGE` idiom to create or update the graph for a [batch of inputs](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/best-practices-content.html#best-practices-content-14). Graph builders and retrievers also identify their operation explicitly, allowing stores such as the RDF / SPARQL contributor to execute a native implementation. The Neptune graph store implementations override the `GraphStore.node_id()` method to ensure that node ids in the code (e.g. `chunkId`) are mapped to Neptune's `~id` reserved property. Alternative graph store implementations can leave the base implementation of `node_id()` as-is. This will result in node ids being mapped to a property of the same name (i.e. a reference to `chunkId` in the code will be mapped to a `chunkId` property of a node).

You use the `GraphStoreFactory.for_graph_store()` static factory method to create a graph store.

Expand All @@ -30,6 +30,7 @@ The lexical-graph supports the following graph databases:
- [Amazon Neptune](/graphrag-toolkit/lexical-graph/graph-store-neptune-db/)
- [Amazon Neptune Analytics](/graphrag-toolkit/lexical-graph/graph-store-neptune-analytics/)
- [Neo4j](/graphrag-toolkit/lexical-graph/graph-store-neo4j/)
- [RDF / SPARQL Stores](/graphrag-toolkit/lexical-graph/graph-store-sparql/)

#### Logging graph queries

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: local-dev
name: graphrag-toolkit-rdf-dev
services:
neo4j-local:
image: neo4j:5.25-community
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ RUN pip install --no-cache-dir \
plotly

# LlamaIndex readers (hard imports in lexical-graph source)
USER root
RUN pip install --no-cache-dir \
llama-index-readers-web \
llama-index-readers-file \
Expand All @@ -42,6 +43,8 @@ RUN pip install --no-cache-dir \
pymupdf

# Build tools for packages requiring C compilation (e.g. lru-dict)
USER root
RUN apt-get update && apt-get install -y --no-install-recommends build-essential && rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y --no-install-recommends build-essential && \
rm -rf /var/lib/apt/lists/* && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"
USER jovyan
Loading