diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..eaffe52 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,166 @@ +--- +title: colada-db architecture +kind: architecture +status: active +updated: 2026-09-06 +owner: danny +verified_by: "N/A — narrative; revisit a couple of times a year, do not sync with code" +--- + +# Architecture + +This is the map. It names files, types and functions and never line numbers, +and it states the invariants that are easiest to miss because they are +*absences* — things the code deliberately does not do. For why a thing is the +way it is, read the ADR it cites. For what changed, `CHANGELOG.md`. + +## Bird's-eye view + +colada-db is a **normalized entity graph in memory, with durability underneath +and agents held at arm's length.** + +``` + app / framework adapter + │ + ┌─────────────────────┼─────────────────────┐ + │ ▼ │ + │ StoreBoundary (boundary.ts) │ ← the only door adapters use + │ │ │ + │ ┌─────────────────┴────────────────┐ │ + │ │ EntityStore (store.ts) │ │ reactive projection — reads never await + │ │ normalize.ts · matcher-view.ts │ │ + │ │ transactions.ts · history.ts │ │ + │ └───────┬──────────────┬────────────┘ │ + │ │ │ │ + │ persist.ts coordinator.ts │ write-behind · sync (on main, unexported) + │ │ │ │ + │ StorageEngine SyncAdapter │ the two ports (ADR-008) + │ memory · idb · restAdapter · │ + │ sqlite (OPFS) wire protocol v1 │ + └───────────────────────────────────────────┘ + │ + packages/mcp (read-only agent surface, ADR-011) + packages/react (useSyncExternalStore binding, ADR-008 §3) +``` + +Memory is the source of UI truth. Everything below it is a **port** with a +narrow contract, and everything beside it is an **edge** that consumes the +boundary and knows nothing about the internals (ADR-008, "boring core, +radical edges"). + +## The core + +**`store.ts` — `EntityStore`.** A `Map` of `entityType:id → EntityRecord`, +reactive through `@vue/reactivity` (standalone, no Vue runtime). `set` / +`replace` / `setMany` / `update` / `remove` / `evict`, refcounted retention +with `gc()`, and one event stream. `getByType` is a projection recomputed from +the map. Every write passes through `runWith({ origin })`, which is how a +`WriteOrigin` gets stamped (ADR-007). + +**`normalize.ts`.** `normalize()` walks a nested payload and lifts every +entity out once, leaving an `EntityRef` in its place; `denormalize()` resolves +refs back with structural sharing. `__typename` is the only auto-detection; +everything else is a `defineEntity` declaration (`types.ts`). + +**`transactions.ts` — `createOptimisticUpdates`.** Optimistic writes with +clear-and-replay rollback, a pre-apply **policy gate** (`useGate`), and a +commit-time last chance (`willCommit`). A veto means the write never touched +the store — `PolicyVetoError` is thrown before apply, not after. + +**`matcher.ts` + `matcher-view.ts`.** A serializable filter AST (`M`, +`parseMatcher`, `evaluateMatcher`) that **fails closed** — anything the +classifier cannot prove maintainable from change events falls to a re-scan +(ADR-009). `createMatcherView` keeps a reference-stable ids array over it +(ADR-010). + +**`history.ts` — `enableHistory`.** A capped field-level change log with +purge-on-remove erasure. **`schema.ts` — `exportSchema`.** The entity registry +as plain JSON, the machine-legible surface an agent reads first. + +**`boundary.ts` — `StoreBoundary`.** `subscribe` (global), `subscribeType`, +`subscribeEntity`, plus synchronous snapshot getters. This is the whole +contract a framework adapter is allowed to depend on; it is the exact shape +`useSyncExternalStore` wants, which is why `packages/react` is thin. + +## Durability + +**`persist.ts` — `enablePersistence(store, { engine })`.** Boot hydration +(`loadAll` or manifest-scoped `loadMany`), then `store.subscribe → dirty set → +debounced engine.writeBatch`. It owns everything engine-agnostic: evict-vs-remove +semantics (ADR-004), the in-flight overlay that keeps pending truth visible +until the engine acknowledges (ADR-015), the optimistic mask (ADR-016), and +graceful degradation — an engine failure disables persistence and the +in-memory store keeps working untouched. + +**`engines/` — the `StorageEngine` port.** `memory` (tests, SSR), `idb` +(default; Safari-hang armor), `sqlite` over OPFS `sahpool` in a worker +(`sqlite-worker.ts`, `sqlite-core.ts`, `sqlite-protocol.ts`; no COOP/COEP +headers). All three are run against one contract kit, +`engine-conformance.ts`, and the persisted format shares one `cdb` prefix and a +reserved `formatVersion` slot (ADR-018). + +**`coalesce.ts`.** The debounced batch flusher shared by persistence and +matcher views. + +## Sync (on `main`, not exported — ADR-022) + +**`sync-types.ts` — `SyncAdapter`.** Three methods, `push` / `pull` / +`subscribe`, server-authoritative and deliberately CRDT-free (ADR-005, +ADR-006). **`coordinator.ts` — `enableSync(store, { adapter })`.** A durable +outbox of locally-committed writes, push with per-mutation verdicts, pull with +version-aware apply, and revert-and-replay for rejected or transformed +mutations. **`rest-adapter.ts`** is the reference adapter speaking +**`wire-protocol.ts`** v1 (`docs/protocol/sync-wire-protocol-v1.md`, ADR-023). +`sync-conformance.ts` and `coordinator-conformance.ts` are the contract kits. + +## The edges + +**`packages/mcp`.** An in-page MCP server over a `StoreBoundary`: schema +resource, query tool (matcher-AST filters, validated fail-closed), optional +history tool. Per-type allowlist; every data result marked untrusted. Runs +over `InMemoryTransport` today (ADR-011). + +**`packages/react`.** `useStoreVersion`, `useEntity`, `useEntities` — +`useSyncExternalStore` over the boundary. `useEntities` caches per +`(boundary, type)` with structural sharing so an unchanged type returns the +same array. + +## Invariants, especially as absences + +- **Architecture Invariant:** reads never await. There is no async read path + anywhere above the engine. A screen that needs cold rows calls `preload` / + `hydrateScope` *before* it renders, not during. +- **Architecture Invariant:** engines never serve reads at runtime. The + `StorageEngine` contract is open / load / writeBatch / close; memory is the + only thing a read touches (ADR-003). A worker query tier would be an ADR-003 + amendment, not an engine method. +- **Architecture Invariant:** the agent surface registers **zero** write + tools. A write attempt is an unknown tool — there is no handler to + misconfigure. Agent write affordances arrive only with a separate, + deliberate guard surface (ADR-011). +- **Architecture Invariant:** types outside the MCP allowlist do not exist to + the agent — absent from the schema, refused by every tool, with refusals + that do not reveal existence. An empty allowlist denies everything. +- **Architecture Invariant:** `evict` has no authority over durability + (ADR-013). Eviction is a memory decision; only `remove` is a semantic delete. +- **Architecture Invariant:** a `WriteOrigin` is stamped by the write channel, + never supplied by the caller through the ordinary API. Origin is attribution + within one trust domain, not authentication. +- **Architecture Invariant:** the sync adapter's arbitration never returns + `"concurrent"` — the server is authoritative and there is no merge (ADR-005, + ADR-006). +- **Architecture Invariant:** nothing in `src/` or `packages/*` imports a + framework runtime or an application-specific type. The Vue reactivity + package is the signal engine, not a Vue dependency (ADR-019 owns the public + read type so it never leaks). +- **Architecture Invariant:** the published surface is asserted, never + printed — `check:api-report`, `check:publish-surface`, `check:pack-manifest` + (ADR-021, ADR-022). What is on npm is exactly what those three say. + +## Where the bodies are buried + +`docs/adr/012`–`017` are one bug family — projection integrity, "the world +held still" — and the philosophy that replaced the patches: order-independence +→ authority → provenance → quiescence → symmetry. Read them before touching +`persist.ts` or `transactions.ts`. `docs/adr/022` lists the six things publish +makes permanent. `LESSONS.md` is the failure log. diff --git a/README.md b/README.md index 4eb50d0..c1d08b3 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ **AI-first, local-first client database.** A normalized reactive entity store with pluggable durability engines — in-memory, IndexedDB, and OPFS SQLite — designed from the cornerstone for a world where AI agents are first-class actors in web applications. +**Your app's state stays on the device. Agents get a scoped, gated, attributed window into it — never the keys.** Types outside the allowlist do not exist to an agent; a vetoed write never touched the store; every write carries its origin; the agent surface registers zero write tools. Each of those is shipped code, not a roadmap item — see [The agent surface](#the-agent-surface-packagesmcp). + > **Status: 0.1.0, the first release.** Early — the API surface is recorded in [`etc/colada-db.api.md`](etc/colada-db.api.md) and changes to it are deliberate, but this is a 0.x package and breaking changes will happen before 1.0. What ships is what is documented here; **there is no sync layer yet** (see the sync bullet below). colada-db was extracted from [`pinia-colada-plugin-normalizer`](https://github.com/Danny-Devs/pinia-colada-plugin-normalizer), which becomes its first framework adapter. ## Install @@ -90,7 +92,7 @@ Each definition must be able to *recognize* its own records: definitions are tri - **AI-first by design — the four trust primitives are in this build.** The committed cornerstone (ADR-007), shipped 2026-07-19: origin tags on every write (`WriteOrigin`, stamped by each write channel — unforgeable through the ordinary write API), a **pre-apply** policy veto gate (`useGate`: a veto means the write never touched the store; commit-time `willCommit` is last-chance and rolls back), a capped queryable history store (`enableHistory`: field-level old→new rows with write ids and origins, purge-on-remove erasure — settled state; settle transactions before logout flows, see the module docs — count + byte bounds), and a machine-legible schema export (`exportSchema`: the registry as plain JSON — the future MCP resource). Each justified by non-AI needs (undo, sync, devtools), each the substrate for agent attribution, policy enforcement, and the agent surface below. Origin = attribution within one trust domain, not authentication. - **Query-driven hydration — memory is a projection, not the whole DB.** Scope manifests (`setManifest`) persist which entities each query/screen needs; `hydration: "manifest"` boots by loading exactly that set via `loadMany` (never a full scan), retained per scope so GC can't evict what a live scope uses. `removeManifest` releases + sweeps; `hydrateScope`/`preload` page durable-but-cold rows back in. Two documented boundaries: **type enumeration reflects the memory projection, not the DB** (cold rows are invisible to any API that walks the store until a scope pulls them in), and **`===` stability ends at evict** — re-hydration materializes new object identity; within-session stability is unaffected because retained entities are never evicted. Without `preload`, first paint on a cold entity shows pending (the synchronous `store.has` check can't see disk). See `docs/design/query-driven-hydration.md`. (DAN-578) - **Live filtered views, two-tier.** `createMatcherView` keeps a reference-stable membership view (ids array, `===`-stable while membership is unchanged) over the serializable matcher AST (ADR-009): validated filters update **purely from change events** — zero query re-runs; closures fall back to coalesced re-scans, always correct. Members are retained while displayed (GC can never evict a live result), and a dev-mode `verifyIntegrity` guard re-scans and self-heals so the fast tier can never silently diverge from re-run truth. Honest boundary: the view's universe is the **memory projection** — durable-but-cold rows are invisible until hydrated (worker-seeded universes are the Stage-2d worker tier's job). See `docs/design/live-matcher-views.md`. (ADR-010, DAN-606) -- **Server-authoritative sync — specified, not yet shipped.** The three-method `SyncAdapter` contract (pull/push/subscribe) is designed and frozen on paper, battle-tested on paper against seven production sync systems, and deliberately CRDT-free. **No adapter ships in this release and nothing sync-related is exported yet** — ADR-006 is still `Proposed`, with implementation scheduled for Stage 3. It is listed here because the durability layer was built to accept it, not because you can call it today. (ADR-005, ADR-006) +- **Server-authoritative sync — built on `main`, not yet in the published package.** The three-method `SyncAdapter` contract (pull/push/subscribe, deliberately CRDT-free) has a coordinator (`enableSync`), a reference `restAdapter`, a documented wire protocol v1 and a durable outbox on `main`, all reviewed and CI-gated. **None of it is exported from the package root yet**, so `npm install colada-db@0.1.0` has no sync. Exporting it changes the public API surface and freezes a wire shape (ADR-022 lines 2 and 5), so it ships as a deliberate 0.2.0 after an adversarial review — not as a side effect of a merge. (ADR-005, ADR-006, `docs/protocol/sync-wire-protocol-v1.md`) - **One reactive graph.** Built on `@vue/reactivity` (standalone — no Vue runtime dependency). Framework adapters share the engine's reactivity instead of shimming a second signal system into it. ## The agent surface (`packages/mcp`) @@ -108,6 +110,10 @@ What it enforces, honestly stated: - **Returned app data is marked untrusted** — in-band envelope (`untrusted: true` + notice) plus `_meta["colada-db/untrusted"]` on results and content blocks. Entity data can originate from servers, other users, or any code with store access: treat it as data, never as instructions. The marking labels the channel; it cannot force a model to comply — pair it with a client/host that honors such labels. - **History honors erasure.** The `read_history` tool (registered only when a history store is provided) serves the capped field-level change log; removed entities' rows are purged, leaving data-free markers only. +## Architecture + +[`ARCHITECTURE.md`](ARCHITECTURE.md) is the map: how the store, persistence, engines, transactions, matcher views, sync, and the agent surface fit together, and the invariants that are easiest to state as absences. + ## Architecture decisions The load-bearing choices live in [`docs/adr/`](docs/adr/) — memory projection over store swap, evict vs delete, sync posture, the SyncAdapter contract, and the AI-first cornerstone. diff --git a/llms.txt b/llms.txt index 8fa81b9..77679e9 100644 --- a/llms.txt +++ b/llms.txt @@ -11,14 +11,24 @@ > root. Framework-agnostic core; adapters consume a two-function subscription > boundary; Vue gets a privileged direct-ref path. -Status: pre-release, unpublished. Extracted 2026-07-19 from the shipped -`pinia-colada-plugin-normalizer` (npm, 0.3.0), which becomes its Vue adapter. +Status: 0.1.0 on npm (published 2026-08-02, MIT), repo public, CI blocking on +`main`. Extracted 2026-07-19 from the shipped `pinia-colada-plugin-normalizer` +(npm, 0.3.0), which becomes its Vue adapter. -What is NOT in this build, stated plainly: no sync. The three-method -`SyncAdapter` contract is specified (ADR-006, still `Proposed`) and the -durability layer was built to accept it, but no adapter ships and nothing -sync-related is exported. `packages/mcp` (the read-only MCP agent surface) -lives in this repo and runs, but is not published to npm. +Positioning, in the one sentence that survives inspection: your app's state +stays on the device; agents get a scoped, gated, attributed window into it — +never the keys. Every clause maps to shipped code (allowlist · pre-apply policy +gate · origin tags · zero write tools on the agent surface). + +What is NOT in the PUBLISHED build, stated plainly: no sync. On `main` the +sync layer exists and is reviewed — `enableSync` coordinator +(src/coordinator.ts), `restAdapter` (src/rest-adapter.ts), wire protocol v1 +(src/wire-protocol.ts, docs/protocol/), durable outbox — but nothing sync- +related is exported from the package root yet, so `npm install colada-db` +gets no sync. Exporting it crosses ADR-022 lines 2 and 5 and is a deliberate +0.2.0 act, preceded by an adversarial review. `packages/mcp` (the read-only +MCP agent surface) and `packages/react` (the `useSyncExternalStore` binding) +live in this repo and run, but are not published to npm. ## Read first - AGENTS.md: how to work on this repo — verify commands, extraction chip state, reading order @@ -39,12 +49,16 @@ lives in this repo and runs, but is not published to npm. - history.ts: enableHistory — capped field-level change log with purge-on-remove erasure - schema.ts: exportSchema — the entity registry as plain JSON (the MCP resource) - transactions.ts also carries the policy gate (useGate, PolicyVetoError) +- coordinator.ts: enableSync(store, {adapter}) — the server-authoritative sync coordinator (ADR-006 rev d); ON MAIN, NOT EXPORTED +- rest-adapter.ts · wire-protocol.ts · sync-types.ts · sync-conformance.ts: the reference SyncAdapter, wire protocol v1, the contract kit; same status - coalesce.ts: createCoalescer — debounced batch flush shared by persist and matcher views - engine-conformance.ts: the shared StorageEngine contract kit every engine is run against - types.ts: EntityStore/StorageEngine/EntityEvent contracts, defineEntity, EntityRegistry (module augmentation) ## Also here - packages/mcp: read-only MCP agent surface (ADR-011). ZERO write tools registered; per-type allowlist; results marked untrusted. In-repo, not on npm. +- packages/react: useStoreVersion / useEntity / useEntities over the adapter boundary (ADR-008 §3). In-repo, private, not on npm. +- ARCHITECTURE.md: the map — how the subsystems above fit, and the invariants stated as absences. ## Verify CI=true pnpm -r test · pnpm -r typecheck · pnpm -r build · pnpm -r lint (all must be green)