A local-first standard for personal information (mail, contacts, calendars): a portable SQLite-plus-blobs store, with a sync layer that keeps it a replica of every source and a search layer that indexes and queries it
One standard in three parts, one store: the format any language with a SQLite binding reads and writes, the engine that keeps it a replica of every source, and the index and query language over it. The canonical part is the schema, the reference statements and the test vectors, so every implementation agrees on the same store. io-pimdir is the reference implementation of the store and the sync layer; the search layer has none yet.
- Features
- Specification
- Using the standard
- Layout
- Status
- AI policy
- License
- Social
- Contributing
- Sponsoring
- Cross-domain: one store for mail, contacts, events, tasks and journals, keyed by media type, with typed summaries per kind and one address table across all of them, so "everything about this person" is one seek.
- Multi-account: several accounts in one store, so a merged view is a query rather than a fan-out. The account groups collections and partitions no identifier.
- Scalable and indexed: hundreds of thousands of items with real secondary indexes, a change feed for anything derived from the store, and keyset pages that cost the same at any depth.
- Portable: one SQLite file, byte-identical across every OS and architecture, with none of the pitfalls of file-per-item layouts.
- Transactional: a whole flag-set change or a multi-item move is one atomic commit a reader never catches half-done.
- Deduplicated: bodies are stored once by content hash, so a message filed in two mailboxes costs one copy.
- Retentive: an item the last source dropped is retained rather than erased, so an upstream expunge never destroys the local copy. Purging is explicit, and restoring costs no network.
- Offline-first: every edit is staged locally and reconciled by a three-way merge against each source, several sources propagating through one item without a cross-merge.
- Searchable: a full-text index over the bodies, calendar occurrences and threads, and a query language every client answers alike.
- Rebuildable: the database is a derived index over the bodies and the remote, and the search index is derived from both, so corruption is survivable by re-sync and a dropped index costs a rebuild.
The standard is a base and two layers, each a normative part written to RFC 2119 with a status of its own, framed by two informative documents:
- OVERVIEW.md, the model: what a store is, the entities, the identifiers, the roles and how sync, retention, the queue and search fit together, with no table, column or statement named. Read it first.
- STORAGE.md, the store, binding by the profile an implementation meets, reader, producer or owner: a SQLite database (the queryable index and mutable state) plus a content-addressed blob directory (the bodies), the per-kind summaries and addresses, the change feed, retention and the action queue.
- SYNC.md, the sync layer: how one or more sources are reconciled through the store, the five verbs, the merge rules and what a connector hands the engine.
- SEARCH.md, the search layer: the derived index beside the store, extraction per kind, calendar time, threads, and the query language every client answers alike.
- GUIDE.md, the implementation guide: the parts' rules as numbered procedures and decision tables naming the statements and vectors at each step, and a conformance checklist.
Either layer may be omitted, and an implementation offering one must conform to it. Neither stands without the store: a sync needs a base per source, what that source last agreed to, or a delete on one side and an add on the other are the same picture, and the bindings are that base; a search needs one index with one meaning of a match across sources, which only the store's summaries, addresses and bodies give. The overview and the guide restate and never rule: where either disagrees with a part, the part wins.
Two properties are worth knowing before reading. Removal is a soft delete: when the last source drops an item, the store keeps the row and its body, hidden from the sync seam and from the live reads, and only an explicit purge deletes it. And the action queue is the write door for every process that does not own the store: a producer appends a kind plus a versioned JSON payload, the owner applies it in append order.
There are three ways to use it, and the right one depends on the part.
- Implement the documents. Any language with a SQLite binding. The store by profile, reader and producer being small; the engine in full; the index and the query language. The vectors say when it is done.
- Use io-pimdir's I/O-free core. The derivations and the five verbs as coroutines: your code answers the storage and remote requests they yield, so the store can be your own SQLite and the network your own transport, in any runtime.
- Use io-pimdir's std client. The whole thing: store, engine and connector seam, behind one handle per profile.
Readers and producers of the store, and clients of the query language, are expected to implement the documents: that is what the format is for. Owning the store, syncing it and indexing it are large, and io-pimdir exists so nobody has to write them twice; SYNC in particular is doable from the document and the vectors, and is where writing from scratch costs the most and gains the least.
OVERVIEW.md the model (informative, read first)
STORAGE.md the store, the base (normative, RFC 2119)
SYNC.md the sync layer (normative, optional)
SEARCH.md the search layer (normative, optional)
GUIDE.md the implementation guide (informative procedures)
migrations/ canonical, forward-only schema migrations (SQL)
storage/ the store's, 0001_init.sql = schema version 1
search/ the search index's, 0001_init.sql = index version 1
queries/ the reference statements, one file per statement named after it
storage/ the store's: read/ the reader's, queue/ the producer's, owner/ the rest
search/ the search index's, prepared with the store attached
vectors/ the normative test data (STORAGE.md §16, SYNC.md §11, SEARCH.md §11)
checks/ what a push checks, needing no implementation: the schema, the statements
through the parts' scenarios, the names, the vectors
flake.nix the toolchain those checks run under
cairn/ the dated history of what the spec did and why (log/), and the audit ledgers (changes/)
AGENTS.md how a contributing agent records a change here
README.md this file
LICENSE-MIT dual license
LICENSE-APACHE
The history follows Cairn, with one deviation while the format is draft: the three documents are the living spec, so cairn/spec/ stays empty rather than restating them, and a landed change is recorded by its log entry alone. AGENTS.md states what ends that.
Draft, pinned as draft-01: the three normative parts carry Status: draft-01 and the git tag draft-01 names the text. An implementation states the draft it conforms to, and a change to a normative part after the tag bumps the number in the three Status lines and here, tagged again; a draft number names a text, never a store. Store schema version 1 and index schema version 1 are defined and stable in shape; the sync and search parts are edited in place between drafts.
The freeze is by profile. STORAGE freezes first, since it is what readers and producers depend on and they are the many; SYNC and SEARCH stay draft behind the reference implementation until their vectors have run against a second implementation. The gate for freezing STORAGE is the reference implementation green on every check and vector in this repository, and the audit ledger under cairn/changes/ with no open item that touches the schema.
While a part is draft, its version is edited in place: a change folds into the migration, the version stays 1, and a store or index created by an earlier draft is recreated rather than migrated. After a freeze, a breaking change to the store bumps the version and ships as a new migration; a change to the index is always a rebuild.
This project is licensed under either of:
- Chat on Matrix
- News on Mastodon or RSS
- Mail at pimalaya.org@posteo.net
Special thanks to the NLnet foundation and the European Commission that have been financially supporting the project for years:
- 2022 → 2023: NGI Assure
- 2023 → 2024: NGI Zero Entrust
- 2024 → 2026: NGI Zero Core
- 2026 → 2027: NGI Zero Commons Fund
This program is part of Pimalaya, free software funded entirely by grants and donations. If you find it useful, consider sponsoring its development:
