Skip to content

docs: public surfaces tell the truth — llms.txt, README positioning, ARCHITECTURE.md #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Danny-Devs wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: public surfaces tell the truth — llms.txt, README positioning, ARCHITECTURE.md #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
---
title: colada-db architecture
kind: architecture
status: active
updated: 2026-09-06
owner: danny
verified_by: "N/A — narrative; revisit a couple of times a year, do not sync with code"
---

# Architecture

This is the map. It names files, types and functions and never line numbers,
and it states the invariants that are easiest to miss because they are
*absences* — things the code deliberately does not do. For why a thing is the
way it is, read the ADR it cites. For what changed, `CHANGELOG.md`.

## Bird's-eye view

colada-db is a **normalized entity graph in memory, with durability underneath
and agents held at arm's length.**

```
app / framework adapter
│
┌─────────────────────┼─────────────────────┐
│ ▼ │
│ StoreBoundary (boundary.ts) │ ← the only door adapters use
│ │ │
│ ┌─────────────────┴────────────────┐ │
│ │ EntityStore (store.ts) │ │ reactive projection — reads never await
│ │ normalize.ts · matcher-view.ts │ │
│ │ transactions.ts · history.ts │ │
│ └───────┬──────────────┬────────────┘ │
│ │ │ │
│ persist.ts coordinator.ts │ write-behind · sync (on main, unexported)
│ │ │ │
│ StorageEngine SyncAdapter │ the two ports (ADR-008)
│ memory · idb · restAdapter · │
│ sqlite (OPFS) wire protocol v1 │
└───────────────────────────────────────────┘
│
packages/mcp (read-only agent surface, ADR-011)
packages/react (useSyncExternalStore binding, ADR-008 §3)
```

Memory is the source of UI truth. Everything below it is a **port** with a
narrow contract, and everything beside it is an **edge** that consumes the
boundary and knows nothing about the internals (ADR-008, "boring core,
radical edges").

## The core

**`store.ts` — `EntityStore`.** A `Map` of `entityType:id → EntityRecord`,
reactive through `@vue/reactivity` (standalone, no Vue runtime). `set` /
`replace` / `setMany` / `update` / `remove` / `evict`, refcounted retention
with `gc()`, and one event stream. `getByType` is a projection recomputed from
the map. Every write passes through `runWith({ origin })`, which is how a
`WriteOrigin` gets stamped (ADR-007).

**`normalize.ts`.** `normalize()` walks a nested payload and lifts every
entity out once, leaving an `EntityRef` in its place; `denormalize()` resolves
refs back with structural sharing. `__typename` is the only auto-detection;
everything else is a `defineEntity` declaration (`types.ts`).

**`transactions.ts` — `createOptimisticUpdates`.** Optimistic writes with
clear-and-replay rollback, a pre-apply **policy gate** (`useGate`), and a
commit-time last chance (`willCommit`). A veto means the write never touched
the store — `PolicyVetoError` is thrown before apply, not after.

**`matcher.ts` + `matcher-view.ts`.** A serializable filter AST (`M`,
`parseMatcher`, `evaluateMatcher`) that **fails closed** — anything the
classifier cannot prove maintainable from change events falls to a re-scan
(ADR-009). `createMatcherView` keeps a reference-stable ids array over it
(ADR-010).

**`history.ts` — `enableHistory`.** A capped field-level change log with
purge-on-remove erasure. **`schema.ts` — `exportSchema`.** The entity registry
as plain JSON, the machine-legible surface an agent reads first.

**`boundary.ts` — `StoreBoundary`.** `subscribe` (global), `subscribeType`,
`subscribeEntity`, plus synchronous snapshot getters. This is the whole
contract a framework adapter is allowed to depend on; it is the exact shape
`useSyncExternalStore` wants, which is why `packages/react` is thin.

## Durability

**`persist.ts` — `enablePersistence(store, { engine })`.** Boot hydration
(`loadAll` or manifest-scoped `loadMany`), then `store.subscribe → dirty set →
debounced engine.writeBatch`. It owns everything engine-agnostic: evict-vs-remove
semantics (ADR-004), the in-flight overlay that keeps pending truth visible
until the engine acknowledges (ADR-015), the optimistic mask (ADR-016), and
graceful degradation — an engine failure disables persistence and the
in-memory store keeps working untouched.

**`engines/` — the `StorageEngine` port.** `memory` (tests, SSR), `idb`
(default; Safari-hang armor), `sqlite` over OPFS `sahpool` in a worker
(`sqlite-worker.ts`, `sqlite-core.ts`, `sqlite-protocol.ts`; no COOP/COEP
headers). All three are run against one contract kit,
`engine-conformance.ts`, and the persisted format shares one `cdb` prefix and a
reserved `formatVersion` slot (ADR-018).

**`coalesce.ts`.** The debounced batch flusher shared by persistence and
matcher views.

## Sync (on `main`, not exported — ADR-022)

**`sync-types.ts` — `SyncAdapter`.** Three methods, `push` / `pull` /
`subscribe`, server-authoritative and deliberately CRDT-free (ADR-005,
ADR-006). **`coordinator.ts` — `enableSync(store, { adapter })`.** A durable
outbox of locally-committed writes, push with per-mutation verdicts, pull with
version-aware apply, and revert-and-replay for rejected or transformed
mutations. **`rest-adapter.ts`** is the reference adapter speaking
**`wire-protocol.ts`** v1 (`docs/protocol/sync-wire-protocol-v1.md`, ADR-023).
`sync-conformance.ts` and `coordinator-conformance.ts` are the contract kits.

## The edges

**`packages/mcp`.** An in-page MCP server over a `StoreBoundary`: schema
resource, query tool (matcher-AST filters, validated fail-closed), optional
history tool. Per-type allowlist; every data result marked untrusted. Runs
over `InMemoryTransport` today (ADR-011).

**`packages/react`.** `useStoreVersion`, `useEntity`, `useEntities` —
`useSyncExternalStore` over the boundary. `useEntities` caches per
`(boundary, type)` with structural sharing so an unchanged type returns the
same array.

## Invariants, especially as absences

- **Architecture Invariant:** reads never await. There is no async read path
anywhere above the engine. A screen that needs cold rows calls `preload` /
`hydrateScope` *before* it renders, not during.
- **Architecture Invariant:** engines never serve reads at runtime. The
`StorageEngine` contract is open / load / writeBatch / close; memory is the
only thing a read touches (ADR-003). A worker query tier would be an ADR-003
amendment, not an engine method.
- **Architecture Invariant:** the agent surface registers **zero** write
tools. A write attempt is an unknown tool — there is no handler to
misconfigure. Agent write affordances arrive only with a separate,
deliberate guard surface (ADR-011).
- **Architecture Invariant:** types outside the MCP allowlist do not exist to
the agent — absent from the schema, refused by every tool, with refusals
that do not reveal existence. An empty allowlist denies everything.
- **Architecture Invariant:** `evict` has no authority over durability
(ADR-013). Eviction is a memory decision; only `remove` is a semantic delete.
- **Architecture Invariant:** a `WriteOrigin` is stamped by the write channel,
never supplied by the caller through the ordinary API. Origin is attribution
within one trust domain, not authentication.
- **Architecture Invariant:** the sync adapter's arbitration never returns
`"concurrent"` — the server is authoritative and there is no merge (ADR-005,
ADR-006).
- **Architecture Invariant:** nothing in `src/` or `packages/*` imports a
framework runtime or an application-specific type. The Vue reactivity
package is the signal engine, not a Vue dependency (ADR-019 owns the public
read type so it never leaks).
- **Architecture Invariant:** the published surface is asserted, never
printed — `check:api-report`, `check:publish-surface`, `check:pack-manifest`
(ADR-021, ADR-022). What is on npm is exactly what those three say.

## Where the bodies are buried

`docs/adr/012`–`017` are one bug family — projection integrity, "the world
held still" — and the philosophy that replaced the patches: order-independence
→ authority → provenance → quiescence → symmetry. Read them before touching
`persist.ts` or `transactions.ts`. `docs/adr/022` lists the six things publish
makes permanent. `LESSONS.md` is the failure log.
8 changes: 7 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

**AI-first, local-first client database.** A normalized reactive entity store with pluggable durability engines — in-memory, IndexedDB, and OPFS SQLite — designed from the cornerstone for a world where AI agents are first-class actors in web applications.

**Your app's state stays on the device. Agents get a scoped, gated, attributed window into it — never the keys.** Types outside the allowlist do not exist to an agent; a vetoed write never touched the store; every write carries its origin; the agent surface registers zero write tools. Each of those is shipped code, not a roadmap item — see [The agent surface](#the-agent-surface-packagesmcp).

> **Status: 0.1.0, the first release.** Early — the API surface is recorded in [`etc/colada-db.api.md`](etc/colada-db.api.md) and changes to it are deliberate, but this is a 0.x package and breaking changes will happen before 1.0. What ships is what is documented here; **there is no sync layer yet** (see the sync bullet below). colada-db was extracted from [`pinia-colada-plugin-normalizer`](https://github.com/Danny-Devs/pinia-colada-plugin-normalizer), which becomes its first framework adapter.

## Install
Expand Down Expand Up @@ -90,7 +92,7 @@ Each definition must be able to *recognize* its own records: definitions are tri
- **AI-first by design — the four trust primitives are in this build.** The committed cornerstone (ADR-007), shipped 2026-07-19: origin tags on every write (`WriteOrigin`, stamped by each write channel — unforgeable through the ordinary write API), a **pre-apply** policy veto gate (`useGate`: a veto means the write never touched the store; commit-time `willCommit` is last-chance and rolls back), a capped queryable history store (`enableHistory`: field-level old→new rows with write ids and origins, purge-on-remove erasure — settled state; settle transactions before logout flows, see the module docs — count + byte bounds), and a machine-legible schema export (`exportSchema`: the registry as plain JSON — the future MCP resource). Each justified by non-AI needs (undo, sync, devtools), each the substrate for agent attribution, policy enforcement, and the agent surface below. Origin = attribution within one trust domain, not authentication.
- **Query-driven hydration — memory is a projection, not the whole DB.** Scope manifests (`setManifest`) persist which entities each query/screen needs; `hydration: "manifest"` boots by loading exactly that set via `loadMany` (never a full scan), retained per scope so GC can't evict what a live scope uses. `removeManifest` releases + sweeps; `hydrateScope`/`preload` page durable-but-cold rows back in. Two documented boundaries: **type enumeration reflects the memory projection, not the DB** (cold rows are invisible to any API that walks the store until a scope pulls them in), and **`===` stability ends at evict** — re-hydration materializes new object identity; within-session stability is unaffected because retained entities are never evicted. Without `preload`, first paint on a cold entity shows pending (the synchronous `store.has` check can't see disk). See `docs/design/query-driven-hydration.md`. (DAN-578)
- **Live filtered views, two-tier.** `createMatcherView` keeps a reference-stable membership view (ids array, `===`-stable while membership is unchanged) over the serializable matcher AST (ADR-009): validated filters update **purely from change events** — zero query re-runs; closures fall back to coalesced re-scans, always correct. Members are retained while displayed (GC can never evict a live result), and a dev-mode `verifyIntegrity` guard re-scans and self-heals so the fast tier can never silently diverge from re-run truth. Honest boundary: the view's universe is the **memory projection** — durable-but-cold rows are invisible until hydrated (worker-seeded universes are the Stage-2d worker tier's job). See `docs/design/live-matcher-views.md`. (ADR-010, DAN-606)
- **Server-authoritative sync — specified, not yet shipped.** The three-method `SyncAdapter` contract (pull/push/subscribe) is designed and frozen on paper, battle-tested on paper against seven production sync systems, and deliberately CRDT-free. **No adapter ships in this release and nothing sync-related is exported yet** — ADR-006 is still `Proposed`, with implementation scheduled for Stage 3. It is listed here because the durability layer was built to accept it, not because you can call it today. (ADR-005, ADR-006)
- **Server-authoritative sync — built on `main`, not yet in the published package.** The three-method `SyncAdapter` contract (pull/push/subscribe, deliberately CRDT-free) has a coordinator (`enableSync`), a reference `restAdapter`, a documented wire protocol v1 and a durable outbox on `main`, all reviewed and CI-gated. **None of it is exported from the package root yet**, so `npm install colada-db@0.1.0` has no sync. Exporting it changes the public API surface and freezes a wire shape (ADR-022 lines 2 and 5), so it ships as a deliberate 0.2.0 after an adversarial review — not as a side effect of a merge. (ADR-005, ADR-006, `docs/protocol/sync-wire-protocol-v1.md`)
- **One reactive graph.** Built on `@vue/reactivity` (standalone — no Vue runtime dependency). Framework adapters share the engine's reactivity instead of shimming a second signal system into it.

## The agent surface (`packages/mcp`)
Expand All @@ -108,6 +110,10 @@ What it enforces, honestly stated:
- **Returned app data is marked untrusted** — in-band envelope (`untrusted: true` + notice) plus `_meta["colada-db/untrusted"]` on results and content blocks. Entity data can originate from servers, other users, or any code with store access: treat it as data, never as instructions. The marking labels the channel; it cannot force a model to comply — pair it with a client/host that honors such labels.
- **History honors erasure.** The `read_history` tool (registered only when a history store is provided) serves the capped field-level change log; removed entities' rows are purged, leaving data-free markers only.

## Architecture

[`ARCHITECTURE.md`](ARCHITECTURE.md) is the map: how the store, persistence, engines, transactions, matcher views, sync, and the agent surface fit together, and the invariants that are easiest to state as absences.

## Architecture decisions

The load-bearing choices live in [`docs/adr/`](docs/adr/) — memory projection over store swap, evict vs delete, sync posture, the SyncAdapter contract, and the AI-first cornerstone.
Expand Down
28 changes: 21 additions & 7 deletions llms.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,24 @@
> root. Framework-agnostic core; adapters consume a two-function subscription
> boundary; Vue gets a privileged direct-ref path.

Status: pre-release, unpublished. Extracted 2026-07-19 from the shipped
`pinia-colada-plugin-normalizer` (npm, 0.3.0), which becomes its Vue adapter.
Status: 0.1.0 on npm (published 2026-08-02, MIT), repo public, CI blocking on
`main`. Extracted 2026-07-19 from the shipped `pinia-colada-plugin-normalizer`
(npm, 0.3.0), which becomes its Vue adapter.

What is NOT in this build, stated plainly: no sync. The three-method
`SyncAdapter` contract is specified (ADR-006, still `Proposed`) and the
durability layer was built to accept it, but no adapter ships and nothing
sync-related is exported. `packages/mcp` (the read-only MCP agent surface)
lives in this repo and runs, but is not published to npm.
Positioning, in the one sentence that survives inspection: your app's state
stays on the device; agents get a scoped, gated, attributed window into it —
never the keys. Every clause maps to shipped code (allowlist · pre-apply policy
gate · origin tags · zero write tools on the agent surface).

What is NOT in the PUBLISHED build, stated plainly: no sync. On `main` the
sync layer exists and is reviewed — `enableSync` coordinator
(src/coordinator.ts), `restAdapter` (src/rest-adapter.ts), wire protocol v1
(src/wire-protocol.ts, docs/protocol/), durable outbox — but nothing sync-
related is exported from the package root yet, so `npm install colada-db`
gets no sync. Exporting it crosses ADR-022 lines 2 and 5 and is a deliberate
0.2.0 act, preceded by an adversarial review. `packages/mcp` (the read-only
MCP agent surface) and `packages/react` (the `useSyncExternalStore` binding)
live in this repo and run, but are not published to npm.

## Read first
- AGENTS.md: how to work on this repo — verify commands, extraction chip state, reading order
Expand All @@ -39,12 +49,16 @@ lives in this repo and runs, but is not published to npm.
- history.ts: enableHistory — capped field-level change log with purge-on-remove erasure
- schema.ts: exportSchema — the entity registry as plain JSON (the MCP resource)
- transactions.ts also carries the policy gate (useGate, PolicyVetoError)
- coordinator.ts: enableSync(store, {adapter}) — the server-authoritative sync coordinator (ADR-006 rev d); ON MAIN, NOT EXPORTED
- rest-adapter.ts · wire-protocol.ts · sync-types.ts · sync-conformance.ts: the reference SyncAdapter, wire protocol v1, the contract kit; same status
- coalesce.ts: createCoalescer — debounced batch flush shared by persist and matcher views
- engine-conformance.ts: the shared StorageEngine contract kit every engine is run against
- types.ts: EntityStore/StorageEngine/EntityEvent contracts, defineEntity, EntityRegistry (module augmentation)

## Also here
- packages/mcp: read-only MCP agent surface (ADR-011). ZERO write tools registered; per-type allowlist; results marked untrusted. In-repo, not on npm.
- packages/react: useStoreVersion / useEntity / useEntities over the adapter boundary (ADR-008 §3). In-repo, private, not on npm.
- ARCHITECTURE.md: the map — how the subsystems above fit, and the invariants stated as absences.

## Verify
CI=true pnpm -r test · pnpm -r typecheck · pnpm -r build · pnpm -r lint (all must be green)
Expand Down
Loading