Knowledge graphs with Python and Neo4j

Working code and a reproducible dataset for the handbook How to build a knowledge graph with Python and Neo4j.

Everything here runs. The dataset is committed rather than generated on your machine, so the numbers in the handbook are numbers you can reproduce exactly, not numbers you have to take on trust.

What is in here

File	What it does
example.py	The complete end-to-end example from the handbook. Creates constraints, loads the teaching data, answers the question. Start here.
data/*.csv	The 75,500 node dataset, committed. This is the data every measurement was taken on.
load_dataset.py	Loads data/*.csv into any Neo4j instance.
verify_dataset.py	Proves your files and your loaded graph match the handbook. Run this before believing any number.
generate_dataset.py	Regenerates the CSVs. You only need this to change the size or the seed.
benchmark.py	The four performance measurements quoted in the handbook.
queries.cypher	Every Cypher block from the handbook, in order.
check_cypher.py	Runs all 39 blocks through EXPLAIN and fails if any is invalid.
validate.py	Runs the nine-step flow from the handbook end to end against a real instance.

Getting started

You need Python 3.9 or newer and a Neo4j instance. The fastest instance to get is Docker:

docker run -d --name neo4j \
  -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/yourpassword \
  neo4j:5.26

Then:

pip install -r requirements.txt

export NEO4J_URI='bolt://localhost:7687'
export NEO4J_PASSWORD='yourpassword'

python3 example.py

If you are using Neo4j Aura instead, take the URI from the credentials file it gives you. It looks like neo4j+s://xxxxxxxx.databases.neo4j.io. If that fails to connect while the browser works, your network is probably intercepting TLS, and neo4j+ssc:// will get you moving.

The dataset

Two datasets, for two different jobs.

The teaching set is fourteen nodes. It lives inside example.py. It is small enough that you can check every answer by eye, which is the whole point when you are learning what a pattern does.

The measurement set is 75,500 nodes and 148,119 relationships. It is in data/. It exists because fourteen nodes cannot demonstrate anything about performance: a missing index costs nothing, and a query plan showing 25 database accesses impresses nobody.

python3 load_dataset.py     # about 9 seconds
python3 verify_dataset.py   # confirms you have the same data
python3 benchmark.py

It fits inside Neo4j Aura's free tier ceiling of 200,000 nodes, so you can run all of this without paying for anything.

Why the CSVs are committed

Because a generator is not reproducible enough.

The first version of this repository shipped only generate_dataset.py, seeded, which produced the same graph every time on the machine it was written on. That is a weaker guarantee than it sounds. random.sample() has changed its internals across CPython releases, so a reader on a different Python version could have built a different graph, run the benchmarks, got different numbers, and reasonably concluded the handbook was wrong.

So there are now two layers:

1. The CSVs are committed and checksummed. verify_dataset.py holds the SHA256 of each file. Nobody needs to run the generator to get the exact data the handbook measured.
2. The generator was rewritten to derive every random decision from random.random() alone, which is stable across CPython versions. Regenerating and diffing the checksums confirms it.

If verify_dataset.py reports a mismatch, re-clone. Do not edit the CSVs by hand.

What the measurements say

All taken on Neo4j 5.26.29 Community in Docker, against the committed dataset, with the client on the same machine.

An index is worth having. Finding one engineer among 50,000:

How	Operator	Database accesses
No label, no index	AllNodesScan	151,002
Label, no index	NodeByLabelScan	100,002
Index	NodeUniqueIndexSeek	3

That is 33,334 times less work. It is not 33,334 times faster: measured wall clock was 35.4 ms against 4.0 ms, about nine times, because a query also pays for planning and returning results. The work ratio holds anywhere. The speed ratio depends on your machine.

Round trips cost more than the work. Writing 1,000 rows one statement at a time took 2,758 ms. The same 1,000 rows through a single UNWIND took 64 ms. Against a managed instance in another city the same comparison was 91,722 ms against 150 ms, because every one of those 1,000 round trips pays the network latency.

Traversal does not care how big the database is. The multi-hop query from the handbook returns 33 engineers in 150 database accesses against 75,500 nodes. The same query against the fourteen-node teaching set costs 25. The graph is 5,000 times larger and the query is unchanged.

Bound your variable length paths. Starting from the most depended-upon service, reach grows 16 times between two hops and six, and the work grows 22 times:

Bound	Services reached	Database accesses
*1..2	30	290
*1..4	133	1,620
*1..6	481	6,388

A note on Community versus Enterprise

Only uniqueness constraints work on Community Edition, which is what the Docker image above gives you. Property existence, node key and property type constraints all need Enterprise, and Aura runs Enterprise, so the same script can succeed on Aura and fail locally:

Neo.DatabaseError.Schema.ConstraintCreationFailed
Property existence constraint requires Neo4j Enterprise Edition

Everything in this repository uses only IS UNIQUE, so all of it runs on Community.

More

I write and record about system design and data infrastructure:

• systemdesign.academy for the full interactive course
• Total Technology Zonne on YouTube, including a Neo4j with Python series

License

MIT. Use it, teach with it, build on it.