Declare your data model. We decide which database engine each part of it lives in, how it is laid out there, and when it should move — and move it while your application keeps running.
Your code never names a table or an engine. That absence is the product: it is what lets the physical schema change underneath you without touching a line of your code.
from datetime import datetime
from decimal import Decimal
from typing import Annotated
from uuid import UUID
import sde
@sde.entity
class User:
id: UUID
email: str
class Meta:
pii = ["email"]
@sde.entity
class Order:
id: UUID
user: sde.Ref[User]
total: Annotated[Decimal, sde.precision(12, 2)]
created_at: datetime
class Meta:
residency = "EU"
@sde.entity
class Event:
id: UUID
name: str
at: datetimeUser and Order are related, so they are placed together and get a row store. Event is joined to
nothing, so it is free to go somewhere built for it. That split — the transactional core in one
engine, the stream nobody joins in another — is the decision most applications get wrong once, at the
start, and never revisit, because moving a live table is a project rather than a decision.
Early. What works today is the whole path, end to end: declaration, canonical model and version, colocation groups, operation shapes, placement maps with signature verification, routing, hashed identifiers, telemetry aggregated into a window document, schema rendered as a value, participation in a migration that runs while the application serves traffic, query plans read back from a live server, and three engine adapters — PostgreSQL, ClickHouse and the orderbook engine. The first two create their own schema and read and write through it.
Two engines is the first point at which any of this means anything. Between a row store and a column store lies the decision most applications get wrong once, at the start, and never revisit; with one adapter there was nothing to choose between.
The third is a different kind of engine and it changed something about this library rather than
adding to it. It stores L2 orderbook depth in a shape fixed in its own C++ source, so there is no
CREATE TABLE to send it and the relationship inverts: for PostgreSQL and ClickHouse you declare a
model and we choose the physical schema, and here the engine has already chosen. A group either is
that shape or it cannot be placed there, and sde.ORDERBOOK_SHAPE is the shape.
Three differences from a general-purpose store are named rather than smoothed over, because a client
planning around this engine needs to know which promises it does not make. It has no transactions.
It does not enforce a key — two writes with the same one both persist, measured, so a read by key
refuses when it finds two rather than answering with one of them. And its unit of work is an
update of N depth levels rather than a row, because level is a price's index inside an update and
not a column the write API accepts; a single-row write can therefore only produce level 0, and any
other value is refused rather than stored at 0 with the read disagreeing with the write.
Smoothing any of those over had only bad forms. Mapping a client's field names onto the engine's would mean guessing which declared field is the price from what it is called, and reasoning from a name is what this library refuses everywhere else.
The pair is also what makes a claim checkable that was previously only stated. save() the same value
through both adapters, read it back from each, and the two results have to be equal — in content and
in Python type. That test found two divergences the moment it existed: a naive datetime landed in
PostgreSQL as 12:00 and in ClickHouse as 10:00, and a timestamptz came back timezone-aware from one
engine and naive from the other, so the same field read from the two could not even be compared. Both
are fixed; the test is python/tests/test_engine_agreement.py, it needs both servers, and CI fails if
it skipped.
Migration works, and it was built last on purpose: one lost row ends a product like this, so it came
after three checkpoints and with a test that deliberately drops writes in order to prove the
verification notices. The work on the data is in these libraries rather than in the control plane,
because copying and comparing rows means reading them — sde.backfill and sde.verify — and what
crosses the boundary is counters, never rows. Verification is a gate rather than a report: reads are
not switched to a copy that does not match.
The Python local cutover operator adds durable execution and crash recovery
for signed generation-bearing packets, including native access changes for existing Python and
TypeScript processes. Staging creates successive fresh copies while preserving
the source and local recovery history; controller handoff and workload qualification are separate.
The opt-in workload qualification runs installed Python and npm
artifacts under mixed traffic and checks scheduled latency, native recovery and acknowledged values.
Session lifetime defines borrowed/owned connections and transaction
ownership for concurrent applications.
The Weather starter provides a local setup, restricted runtime clients in
Python/TypeScript, measured telemetry, operator handoffs and an ownership-checked reset. It is
unreleased; the runbook uses built artifacts and distinguishes this demo from production qualification.
What does not exist yet is a library in any other language: java/ and rust/ are the next two and
neither directory is here. The day one of them appears, the test over this page fails until the row
below has been rewritten.
| Library | Tier | Status |
|---|---|---|
python/ |
0 and 1, plus 2 for PostgreSQL, ClickHouse and the orderbook engine, plus hashing | reference implementation |
typescript/ |
0, 1 and 2 for PostgreSQL and ClickHouse, plus hashing | passes the same vectors, byte for byte |
java/, rust/, then C#, Go, Kotlin, PHP, Ruby |
— | contributions welcome; the contract now has two implementations, which is what made it safe to invite them |
That qualification used to say something different, and the change is the point of it. Until
6 September 2026 the telemetry/, schema/ and migration/ vector sets named in
docs/format-contract.md §10 did not exist, so Tier 1 and Tier 2 were backed by the reference
implementation's own tests and by slices against real servers — which verifies that it works, not
that a second implementation would agree with it. That cost nothing while one library claimed those
tiers, and the paragraph said what would happen on the day a second one did: the vectors come
before the claim does. TypeScript reached Tier 2 on that day, so all three sets were written first.
Closing the gap found five defects in the implementation that had claimed those tiers unchecked, and that is the honest argument for the whole exercise. Each is described in the contract, with the family that found it.
The table above is this repository's layout. The list of what each library supports, and who fixes
it when it breaks, is docs/implementations.md, and that is the one to
read before depending on any of it.
You declare entities, relations, and four invariants. Everything else about storage is ours.
The four exist because no amount of watching traffic reveals them:
| Declaration | Why traffic cannot tell us |
|---|---|
atomic_with |
that two entities must commit together is a business rule, not a pattern |
residency |
where data may legally live is not visible in a query |
pii |
which column is personal data determines retention and what may be denormalised |
cost_ceiling |
your budget is not in your workload |
If a future feature needs a fifth declaration, that is a change to what the product promises rather than a new configuration option, and it will be argued about as one.
These libraries are Apache-2.0 and free. The control plane — the placement planner, the scoring model, the migration orchestrator — is private, and that is the part you pay for.
Which means the boundary runs in both directions. Nothing from the planner is here, and nothing that touches your data is anywhere else. That second half is why this repository exists: "we never see a row" stops being a promise you have to accept and becomes something you can check.
Three things you can verify without asking us:
- We are not in your data path.
routing.pyis a dictionary lookup and three conditions. There is no code path that sends a query through us, so our outage cannot be your outage. - Telemetry carries no values. An operation shape is assembled from the structure of a call, not from a query string, so there is nowhere for a value to come from. Compare that with the SQL route, where you receive a string full of literals and strip them out with a parser you hope is complete.
- It works without an account. Hand-write a placement map, point the library at it, and everything runs — no key, no network, no account. An unsigned map is valid. That is a supported mode, tested as the default in our own suite, and the honest answer to what happens if you stop paying us.
What is refused is a map that claims to be ours, by carrying a signature, when there is no key to check the claim against. An unverifiable claim is worse than no claim.
One more refusal is worth knowing about, because it means the library keeps a little state. A
signature says a document is authentic and says nothing about whether it is current — a signed map
for version 3 verifies correctly forever. So a signed map is refused if it is older than one
already applied against your engines, and remembering which one that was needs a table:
sde_map_state, append-only, in your own engines, holding a map version and a timestamp and nothing
else. Equal is fine; restarting is ordinary. session.rollback_protection says whether it is in
force, because a guarantee whose state you cannot read is one you have to take on trust — and for an
engine whose schema is fixed in its own source there is nowhere to keep it, so the answer there is
unavailable rather than a pretence. An unsigned map is never checked: it is your document, and
in the no-account mode this costs nothing, creates nothing and queries nothing.
Four libraries computing the same model version is not a nice property, it is the difference between a working product and a broken one. If Python and Java disagree, the same model arrives at the control plane as two models, gets two placements, and half a fleet writes to a different set of tables. Nothing about that fails at compile time.
So the encoding is specified at the byte level in docs/format-contract.md
— UTF-8, keys NFC-normalised then sorted by code point, no insignificant whitespace, minimal escaping,
no float literals, a closed type vocabulary so that Decimal and BigDecimal land on the same bytes.
And conformance/ holds the vectors — 258 of them, in ten families — that every
library runs in its own test runner, so a divergence is a red test for whoever caused it rather than
an operation written to the wrong engine in production.
conformance/vectors/model/001-single-entity was written by hand from the document rather than
generated, which makes it the one vector that proves the document says enough to implement from. So
was every vector under conformance/vectors/canonical/.
Writing TypeScript against the contract, rather than translating Python into it, found a divergence worth the entire exercise. JavaScript compares strings by UTF-16 code unit; the contract requires code point order. Those agree for everything in the Basic Multilingual Plane, so no test written with Latin or CJK identifiers can see the difference — and they disagree above U+FFFF, where an astral character is a surrogate pair starting at 0xD800 and therefore sorts before U+E000 instead of after. One field name like that and two libraries produce two versions of one model.
Then mutation testing found that the first vector written for it did not actually cover the bug. Every
object key in the model IR is fixed ASCII, so swapping the object-key comparator for a naive sort
passed the whole suite; field names reach the IR as array elements, through a different comparator. The
canonical/ vectors exist to close that, and both call sites are now verified by deliberately breaking
them and watching the suite go red.
The same experiment, done deliberately: on 6 September 2026 a Tier 0 library was written in Go, from the contract and the vectors alone, to find out whether that document really is sufficient to implement from without asking us. All 49 vectors passed on the first run, so the encoding half of the claim held. Everything the vectors did not reach did not: six defects in these two libraries, the worst of them a placement map whose groups were validated in the document's own key order, so one document refused differently depending on how a JSON parser handed the keys over — and the vector that was supposed to cover it failed in Go, whose maps iterate in a randomised order, in 5 of 20 runs.
Eleven vectors came out of it. The Go code did not: it was a measurement, not a library, and a fourth
implementation nobody keeps working is worse for a client than none.
docs/implementing.md has the findings, the order to build a library in, and
a table of what each language does to you. docs/implementations.md is the
list of which libraries exist, what each one really does, and who fixes it when it breaks.
The same experiment one day later and one tier further up: Rust, Tier 0 and Tier 1 and Tier 2, from the contract and the vectors alone. The day before, those two upper tiers had gained vectors for the first time, and a claim is worth auditing on its first day rather than its hundredth. Two hundred and thirty-four assertions, one red — and the red one was a misreading of a refusal rather than a defect in it.
Rust was not chosen only for having no library here. Its traits are dispatched statically, so a capability cannot be a missing method, and the reference answers "can this engine keep the bookkeeping" by looking for members. That question does not exist in this language, which is what makes it the one that finds the assumption. Java, C# and Go with non-optional interfaces have the same shape, and the capability has to be data on the value — which §7 of the contract already required.
Ten findings, and the worst of them was an escaper. ClickHouse reads a backslash inside a
backtick-quoted identifier as an escape introducer; this library escaped the backtick and left the
backslash alone. A field named `a\nb` therefore reached the server as a column called a, a
newline and b — a different name, accepted in silence — while `back\slash` survives
untouched, which is what makes the defect look absent. It was measured by reading the name back out
of system.columns, because the statement is accepted either way. No vector in schema/ carried a
delimiter inside an identifier, so the escaping could be removed in its entirety and the family
stayed green. PostgreSQL is the reverse — it doubles the quote, leaves the backslash literal, and
backslash-escaping the quote there is a syntax error — so there is no escaper for the two dialects
to share.
Two of the ten were defects in a live test rather than in a library, and both were about reach: one
statement of the schema/ family had never executed, because an earlier vector had created a
table of that name and CREATE TABLE IF NOT EXISTS never parsed the body, and the test stopped at
"the server accepted it", which a wrong escaper passes. The rest are in
docs/implementing.md, including two percentile conventions in one document,
zero-row results from failed reads averaged into cardinality, and partition_by — in the format
from the beginning, emitted by the control plane, rendered by nobody, and now refused on both sides.
The Rust code is not in this repository either, for the same reason the Go code is not.
docs/failure-semantics.md, one row per failure: the engine not
listening, a host that accepts the socket and says nothing, a connection cut under an operation, a
write during a migration, a map from the wrong model, us being unreachable, your subscription
lapsing. Each row says what your call does, what is retried, what can be lost, and which test pins
it. Written before the first sale rather than after the first incident, and two of the three
connection failures it documents turned out to be defects rather than gaps: neither adapter bounded
opening a connection, so a silent host hung the caller's request path.
cd python && python -m venv .venv && .venv/bin/pip install -e '.[dev,signed,postgres]' && cd ..
cd typescript && npm install && cd ..
make pg-up && make checkmake check runs both languages. It does not stop at the first failure across them on purpose: if
Python and TypeScript have both drifted, you want to see both, because the fix is usually in the
contract rather than in either library.
The integration slices run against real servers rather than fakes. A fake would agree with whatever this library believes about types, quoting and transactions, which is exactly the set of beliefs worth checking.
The orderbook slice is the exception, and the exception is split rather than waived. That engine's Python client is not on PyPI and its shared library is built from C++, so it cannot run everywhere. Everything the adapter decides — the shape check, the level refusal, the refusal on a duplicate key, unknown-not-zero for a sequence number — happens before the client library is called and is tested against a fake, which runs everywhere. What a fake cannot check is whether the engine still behaves as measured, so four measurements are asserted against the engine itself:
git clone https://github.com/Smart-Data-Engines/low-cost-and-low-latency-orderbook-dbengine ../ob
cmake -S ../ob -B ../ob/build && cmake --build ../ob/build -j"$(nproc)"
OB_LIB_PATH=$PWD/../ob/build/liborderbook_shared.so PYTHONPATH=$PWD/../ob/python \
SDE_ORDERBOOK=1 python/.venv/bin/python -m pytest python/tests/test_orderbook_slice.pyIf the engine changes, that file fails and the fake stops describing something true — which is the failure mode a fake normally hides.
CONTRIBUTING.md, and the short version: read the format contract, pass the Tier 0
vectors in your own runner, be honest about which tier you reach, and write an API that is idiomatic in
your language rather than Python transliterated.
Apache-2.0. See LICENSE and NOTICE.
Batch operations: logical bulk writes document save_many / saveMany,
bounds, source/copy ordering and uncertain outcomes.
Logical reads and exact summaries cover scan, count and summarize, including portable keyset pages, typed values and explicit failure semantics.
Engine credentials and verified TLS configuration: connection guide.