Skip to content

Eth-Interchained/nedb

BranchesTags

Repository files navigation

NEDB

Content-addressed Merkle DAG · Hash-chained · Time-traveling · Bi-temporal · Causally-provable embedded database.

Replay-protected · idempotent · relational · filterable · sortable · searchable · concurrent. One Rust core → ships to PyPI and npm from a single source.

PyPI npm Tests nedb-engine-client PyPI nedb-engine-client npm

Studio → studio.interchained.org · nedb.aiassist.net

NEDB v2.0.36 — Production Stable

Current stable: 2.0.36 — Cross-platform native wheels shipping nedbd-v2 binary inside pip install nedb-engine. Linux + Windows wheels built on GitHub Actions; macOS arm64 + x86_64 wheels built on Codemagic (M2 Mac Minis). All four platforms publish from one v* tag.

NEDB v2 replaces the append-only log (AOF) with a content-addressed Merkle DAG. Every document version is an immutable, BLAKE2b-verified object. Nothing is ever overwritten. As of v2.0.36, restarts after the first open are O(1) warm starts (driven by a MANIFEST of seq + Merkle head), the cold scan is deferred so the daemon accepts connections immediately, and a new GET /events SSE endpoint streams scan progress + per-write events live.

# Run the v2 DAG engine — ships inside pip install nedb-engine
nedbd --dag --data ./data
# or
NEDBD_DAG=1 NEDB_TMK=<32-byte-hex> nedbd --data ./data

curl http://127.0.0.1:7070/health
# {"ok":true,"version":"2.0.36","service":"nedbd","engine":"dag","startup_ready":true,"encrypted":true}

# Tail the live event stream (new in v2.0.36)
curl http://127.0.0.1:7070/events
# event: scan   data: {"objects":730000,"of":1310703,"rate":21043,"eta_s":28}
# event: ready  data: {"seq":1310703,"head":"b2:9c14e07a…"}
# event: write  data: {"seq":1310704,"coll":"beliefs","head":"b2:7af3c11e…"}
Property v2 DAG v1 AOF
Uncorruptable (atomic writes, hash-verified reads) ✅ ⚠️
O(1) warm start via MANIFEST (no scan, no replay) ✅ ❌
Deferred cold scan (socket open immediately) ✅ ❌
O(1) incremental Merkle head (never recomputed) ✅ ❌
Parallel writes (no global lock) ✅ ❌
BLAKE2b Merkle head on every response ✅ ❌
IdIndex sharded across 256 subdirectories ✅ ❌
TCP_NODELAY (no 40–200 ms loopback Nagle delay) ✅ ❌
GET /events SSE log stream ✅ ❌
Tombstone deletes (history preserved) ✅ ✅
Auto-migrates v1 AOF → v2 DAG on startup ✅ —
Same HTTP API — Vision, Studio, all clients unchanged ✅ ✅

v1 AOF engine is still shipped and unchanged — nedbd (no flag) runs v1.

Production status: vision.interchained.org is live on v2.0.36 — 1,310,703 sequences indexed in the Vision database, AES-256-GCM encrypted at rest, at block height 620,989.

What makes NEDB different

Every database stores what. NEDB stores what, when, when it was true, and why — all sealed in a cryptographic hash chain that proves none of it was tampered with.

Capability NEDB SQLite Redis MongoDB
Hash-chained tamper evidence ✅ ❌ ❌ ❌
Time-travel reads (AS OF seq) ✅ ❌ ❌ ❌
Bi-temporal (VALID AS OF date) ✅ ❌ ❌ ❌
Causal Write Provenance ✅ ❌ ❌ ❌
Replay-protected idempotent writes ✅ ❌ ❌ ❌
SQL + Redis + MongoDB adapters ✅ — — —
Concurrent group-commit daemon ✅ ❌ ✅ ✅
At-rest AES-256-GCM encryption ✅ ❌ ❌ —

Install

pip install nedb-engine      # Python ≥ 3.8 — pure-Python + optional Rust native wheel
npm install nedb-engine       # Node ≥ 16   — napi-rs prebuilt binaries

Python — 5-minute tour

from nedb import NEDB

db = NEDB("./mydata")          # durable: every op is AOF-logged, fsync'd, and hash-chained
# db = NEDB()                  # or in-memory

db.create_index("users", "status", "eq")
db.create_index("users", "bio",    "search")

db.put("users", "alice", {"name": "Alice", "age": 31, "status": "active", "bio": "rust hacker"})
db.put("users", "bob",   {"name": "Bob",   "age": 24, "status": "active", "bio": "python dev"})

# NQL: WHERE + ORDER BY + LIMIT + SEARCH + TRAVERSE + GROUP BY
db.query('FROM users WHERE status = "active" ORDER BY age ASC')
db.query('FROM users SEARCH "rust"')
db.query('FROM users GROUP BY status COUNT')

# Time-travel — AS OF any past sequence
snap = db.seq
db.put("users", "alice", {"name": "Alice", "age": 32, "status": "retired"})
db.get("users", "alice", as_of=snap)          # → age 31, status active

# Bi-temporal — VALID AS OF any past date
db.put("policy", "rate_2024", {"pct": 5.0}, valid_from="2024-01-01", valid_to="2024-12-31")
db.put("policy", "rate_2025", {"pct": 6.0}, valid_from="2025-01-01")
db.query('FROM policy VALID AS OF "2024-06-15"')   # → rate 5.0

# Causal Write Provenance — why did this write happen?
db.put("inputs", "msg_1", {"text": "user prefers dark mode"})
seq_msg = db.seq
db.put("beliefs", "dark_mode", {"value": True},
       caused_by=[seq_msg], evidence="user_message", confidence=0.95)
db.query('FROM beliefs WHERE _id = "dark_mode" TRACE caused_by')   # → msg_1
db.query('FROM inputs WHERE _id = "msg_1" TRACE caused_by REVERSE') # → dark_mode

# Relations + graph traversal
db.link("users:alice", "follows", "users:bob")
db.query('FROM users WHERE _id = "alice" TRAVERSE follows')

# Hash-chain integrity
assert db.verify()             # cryptographic proof — no tampering

# SQL, Redis, MongoDB compatibility adapters
from nedb import sql_exec, RedisCompat, MongoClient
sql_exec(db, "SELECT * FROM users WHERE status = 'active' ORDER BY age DESC")
r = RedisCompat(db); r.execute("HSET", "user:1", "name", "Alice")
MongoClient(db)["users"].find({"status": "active"}).sort("age", -1).to_list()

Redis layer-2 — wrap_redis()

Already running on Redis? Wrap your connection in one line and gain NEDB features alongside your existing Redis app — no migration required.

import redis, json
from nedb import wrap_redis

r = wrap_redis(redis.Redis("localhost", 6379), db_name="rideshare")

# Step 1 — register: map Redis key globs to NEDB collections (chainable)
(r.nedb
 .register("driver:*", collection="driver", value_parser=json.loads)
 .register("trip:*",   collection="trip",   value_type="hash")
)

# Step 2 — backfill: import all existing Redis data into NEDB in one pass
imported = r.nedb.backfill()           # → int (keys imported)

# Step 3 — shadow: all future r.set/hset/... auto-chain into NEDB
r.nedb.shadow_writes = True

# ─── Alice's app keeps running — zero changes ───────────────────────────
r.set("driver:d1", json.dumps({"name": "Bob", "status": "active"}))   # ← shadowed
r.hset("trip:t1", mapping={"status": "en_route", "driver_id": "d1"})  # ← shadowed

# ─── New features available on the same connection ──────────────────────
r.nedb.query('FROM driver WHERE status = "active" ORDER BY lat ASC')
r.nedb.verify()       # → True  (every write chain-verified)
r.nedb.head()         # → 64-char BLAKE2b commitment hash

Isolation guarantee: NEDB never writes to Alice's namespace. It owns only:

Key Type Purpose
nedb:{db_name}:oplog Redis Stream append-only op log
nedb:{db_name}:snapshot Redis Hash checkpoint
nedb:{db_name}:meta Redis Hash index config

See examples/fakeredis_demo.py for a full local demo (no Redis server needed).

Node.js

import { NedbCore } from "nedb-engine";

const db = new NedbCore();               // in-memory
// const db = NedbCore.open("./data");   // durable

db.createIndex("users", "status", "eq");
db.put("users", "alice", JSON.stringify({ name: "Alice", age: 31, status: "active" }));

// Time-travel
const snap = db.seq();                   // BigInt
db.put("users", "alice", JSON.stringify({ name: "Alice", age: 32, status: "retired" }));
JSON.parse(db.getAsOf("users", "alice", snap)).age;  // → 31

// Full NQL
const rows = db.query('FROM users WHERE status = "active" ORDER BY age ASC');
rows.map(r => JSON.parse(r));

// Tamper evidence
db.verify();   // → true
db.head();     // → 64-char BLAKE2b commitment hash
db.seq();      // → BigInt

nedbd — the concurrent server daemon

nedbd runs NEDB as a long-lived process with an HTTP/JSON API and an optional RESP2 wire protocol. Built on a single-writer group-commit sequencer — parallel reads, batched durable writes, one hash-chain per database, zero write-write races.

nedbd                                     # :7070, data ./nedb-data (v1 AOF engine)
nedbd --dag --data ./data                 # v2 DAG engine (or NEDBD_DAG=1)
NEDBD_RESP2_PORT=6380 nedbd               # also speak RESP2 (redis-cli compatible)
nedbd --log-level 2                       # 0=errors 1=requests 2=deploy 3=verbose

# Live event stream (new in v2.0.36) — SSE: scan progress, ready, per-write head
curl http://127.0.0.1:7070/events

Startup modes (v2.0.36)

  • Warm start — every restart after the first open reads the MANIFEST file and restores seq + Merkle head in O(1). No scan, no replay, independent of dataset size. Boots in milliseconds.
  • Cold start — first open of an existing dataset spawns the integrity scan in a background thread and accepts connections immediately. Reads serve instantly from the content-addressed DAG; writes return HTTP 503 startup in progress until the startup_ready gate flips. Progress (objects, rate, ETA) streams over GET /events.

Environment variables

Variable Default Description
NEDBD_DAG 0 Set 1 to launch the v2 DAG engine (nedbd-v2). Same as --dag.
NEDBD_HOST 127.0.0.1 Bind address. v2.0.36 defaults to loopback (was 0.0.0.0) — security hardening fix. Set explicitly to 0.0.0.0 to expose.
NEDBD_PORT 7070 HTTP bind port.
NEDBD_TOKEN unset Optional bearer token; required on every /v1/* request when set.
NEDB_TMK unset 32-byte hex AES-256-GCM at-rest encryption key.
NEDBD_DATA ./nedb-data Root directory. v2 creates dag/, IdIndex sharded across 256 subdirectories, and a small MANIFEST file. 
# Create a database with seed data and relations
curl -X POST :7070/v1/databases -d '{
  "name": "shop",
  "init": {
    "indexes": [["users","status","eq"]],
    "seed": {"users": [{"_id":"u1","name":"Alice","status":"active"}]},
    "links": [["users:u1","buys","orders:o1"]]
  }}'

# Query (full NQL including time-travel and bi-temporal)
curl -X POST :7070/v1/databases/shop/query \
  -d '{"nql":"FROM users WHERE status = \"active\" ORDER BY name ASC"}'

# Verify the hash chain
curl :7070/v1/databases/shop/verify

# MongoDB-compatible endpoint
curl -X POST :7070/v1/databases/shop/mongo \
  -d '{"collection":"users","op":"find","filter":{"status":"active"},"limit":10}'

From redis-cli — no Redis installation needed:

redis-cli -p 6380 SELECT shop
redis-cli -p 6380 SELECT shop EVAL 'FROM users SEARCH "alice"' 0
redis-cli -p 6380 SELECT shop EVAL 'FROM users AS OF 10 WHERE status = "active"' 0
redis-cli -p 6380 SELECT shop EVAL 'FROM beliefs TRACE caused_by' 0

NQL — the NEDB Query Language

FROM <collection>
  [ AS OF <seq> ]                            transaction time (when was it written?)
  [ VALID AS OF "<date>" ]                   valid time (when was it true in the world?)
  [ WHERE <field> <op> <value> (AND ...) ]   op: = != < <= > >=
  [ SEARCH "<text>" ]                        full-text search
  [ ORDER BY <field> [ASC|DESC] ]
  [ TRAVERSE <relation> ]                    graph traversal
  [ TRACE caused_by [REVERSE] ]              causal provenance (why? / what did this cause?)
  [ LIMIT <n> ]
  [ GROUP BY <field> [COUNT|SUM f|AVG f|MIN f|MAX f] ]

Combine both time axes:

# What did the system know at seq 200 about what was true on 2024-02-15?
db.query('FROM policy AS OF 200 VALID AS OF "2024-02-15"')

Performance

v2 DAG Rust server (v2.0.36, Intel iMac — 10k writes / 100k reads / 30k objects, AES-256-GCM on):

Operation Throughput p50 p99
Sequential writes 418 ops/s 2.3 ms 3.3 ms
Point-lookup reads 478 ops/s 2.0 ms 3.0 ms
ORDER BY queries 489 ops/s 1.8 ms 4.3 ms
Batch writes (500 ops/req) 1,104 ops/s 0.9 ms 1.2 ms
Tamper-verify (30k objects) ~21,000 BLAKE2b/sec — 1.38 s total

p99 latencies hold because of TCP_NODELAY on the axum listener — without it macOS loopback adds the Nagle algorithm's 40–200 ms delay on small writes.

v1 Python server (baseline — single-threaded AOF):

Operation Throughput p99 latency
Sequential PUT ~23/s 44 ms
Concurrent PUT (16 workers) ~92/s 48 ms
Batch PUT (500 ops/request) ~520 ops/s 1.9 ms/op
Point-lookup read (NQL) ~23/s 44 ms
Rust napi PUT (FFI) ~70K/s —
Rust napi GET (FFI) ~330K/s —

Reproduce with the included benchmark:

NEDBD_DAG=1 nedbd --data /tmp/perf &
python3 tests/test_dag_perf.py --n 10000 --reads 100000

Architecture

            ┌──────────────────────────────────────────────────────────┐
  put/del → │  OpLog  (BLAKE2b hash chain · per-client nonce ·          │ ← single source of truth
  link      │          idempotency keys · causal provenance fields)     │
            └───────────────┬──────────────────────────────────────────┘
            deterministic fold │ (state = pure function of the log)
     ┌──────────────┬──────────┴──────┬───────────────┬────────────────┐
     ▼              ▼                 ▼               ▼                ▼
MVCC store     Relations          Indexes         CauseMap          BlobStore
(time-travel)  (graph+AS OF)      eq/ord/search   (reverse index)   (Cascade CDC)

                     ┌─────────────────────────────────┐
  Thread-safe →      │  Sequencer (group-commit)         │ ← single writer, parallel readers
                     │  — one committer thread/db        │
                     │  — batch fsync                    │
                     └─────────────────────────────────┘

Compatibility adapters:  SQL  ·  Redis  ·  MongoDB
Wire protocols:          HTTP/JSON  ·  RESP2
Encryption:              AES-256-GCM at-rest (TMK/DEK double-envelope)

nedb-client — lightweight HTTP client

Connect to any running nedbd instance from Python or TypeScript without embedding the engine:

pip install nedb-engine-client          # async Python
npm install nedb-engine-client   # TypeScript / Node.js 18+
from nedb_client import NedbClient

async with NedbClient("http://127.0.0.1:7070", db="mydb") as db:
    await db.put("blocks", "618000", {"height": 618000})
    rows = await db.query("FROM blocks ORDER BY height DESC LIMIT 10")
    head = await db.head()    # BLAKE2b Merkle root — changes on every write
    ok   = await db.verify()  # tamper-evidence check across all objects
import { NedbClient } from "nedb-engine-client";
const db = new NedbClient({ url: "http://127.0.0.1:7070", db: "mydb" });
await db.put("blocks", "618000", { height: 618000 });
const rows = await db.query("FROM blocks LIMIT 10");

Repo layout

python/nedb/        reference engine (pure Python — always-works baseline)
rust/
  nedb-core/        v1 production Rust engine (shared by both runtimes)
  nedb-py/          maturin PyO3 binding → PyPI native wheels
  nedb-node/        napi-rs binding → npm native addons
  nedb-v2/          v2 DAG engine (tokio + axum + BLAKE2b DAG)
client/
  python/           nedb-client — async Python HTTP client (pip install nedb-engine-client)
  node/             nedb-client — TypeScript HTTP client  (npm install nedb-client)
tests/              engine + concurrent + causal + bitemporal + deploy + perf benchmarks
examples/           resp2_python.py  resp2_demo.sh
docs/               index.html  reference.html  SPEC.md

Roadmap

  • Hash-chained append-only log — tamper evidence, replay protection, idempotency
  • MVCC time-travel — AS OF seq
  • Bi-temporal — VALID AS OF "date" (transaction time + valid time)
  • Causal Write Provenance — caused_by, evidence, confidence, TRACE
  • Durable AOF persistence + snapshot checkpoints
  • Concurrent group-commit sequencer (nedbd, 15K writes/s under load)
  • AES-256-GCM at-rest encryption (TMK/DEK double-envelope)
  • SQL / Redis / MongoDB compatibility adapters
  • RESP2 wire protocol (redis-cli / redis-benchmark compatible)
  • Rust native core — napi-rs (npm) + maturin PyO3 (PyPI)
  • Self-healing AOF — auto-truncates corrupt tail on startup, never hangs
  • v2 DAG engine — content-addressed Merkle DAG, atomic writes, instant cold start
  • nedbd --dag — one flag switches to v2 Rust engine; v1 untouched
  • BLAKE2b Merkle head — tamper-evident root on every response
  • Tombstone deletes — history preserved in DAG, live id removed from index
  • Auto-migration — v1 AOF → v2 DAG on first --dag startup
  • nedb-client — async Python + TypeScript HTTP client (pip/npm install nedb-client)
  • Intel Mac support — native wheels for aarch64 + x86_64 Apple Darwin
  • In-memory DAG mode — Db::in_memory() for zero-disk ephemeral sessions
  • PyO3 + napi-rs bindings updated to v2 DAG API
  • NEDB Studio DAG mode toggle
  • Merkle inclusion proofs — prove a document existed at a specific time to a third party
  • Git-style branching — fork database state, experiment, merge or discard
  • Agent Memory SDK — Memory.remember() / Memory.recall() / Memory.trace()
  • Live query subscriptions (SSE) — push diffs when query results change

NEDB Studio

Prompt-to-database scaffolding GUI with schema graph, NQL console, time-travel slider, causal provenance panel, and MongoDB/SQL/Redis tabs. Deploy from a description, query live data, edit inline.

studio.interchained.org · github.com/aiassistsecure/nedb-studio (GPLv3)

Repos

Repo Description
aiassistsecure/nedb Source — engine, Rust core, CI
aiassistsecure/nedb-studio Studio UI (GPLv3)

Packages: PyPI nedb-engine · npm nedb-engine

License

See LICENSE file. · © INTERCHAINED, LLC — interchained.org

Authors

Built by Mark Allen Evans Jr. (INTERCHAINED, LLC) with Claude Sonnet 4.6 on Hyperagent.

"Take one idea, turn it into an LP, then an app, then a system, then a platform, then infrastructure that is irreplaceable."

Built with Hyperagent AiAssist

About

NEDB - versioned, self-compressing, time-traveling embedded database

Resources

Readme

License

View license
Activity

Stars

0 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases 10

v2.1.0 Latest
Jun 17, 2026
+ 9 releases

Packages

 
 
 

Contributors

Languages