Skip to content

feat: add preload hybrid db #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
quolpr merged 3 commits into from
Aug 24, 2026
Merged

feat: add preload hybrid db #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
266b765
feat: add preload hybrid db
quolpr Aug 24, 2026
6281b6f
fix: improve external merge
quolpr Aug 24, 2026
83dfe97
fix: fix after review
quolpr Aug 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .changeset/green-indexes-rest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
"@will-be-done/hyperdb": minor
---

Add `PreloadedHybridDB`, which preloads every declared index with ID-only leaves
and batch-hydrates missing entity rows through the built-in `byId` `uniqhash`.
Secondary `uniqhash` indexes are preloaded as value-to-ID pointers, while shared
hash-index transactions provide copy-on-write commit and rollback behavior.
Add `externalStorageMergeTrait` for changesets already persisted by another
runtime sharing the primary. Their normal merge operations update the preloaded
snapshot, notify subscribers, and persist external inserts idempotently.
35 changes: 26 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ to strain:
possible and load missing ranges from the primary store on demand. Writes
update the cache first so the UI can respond immediately, then flush to the
primary store in order.
- **Preloaded indexes without preloaded rows.** `PreloadedHybridDB` loads every
declared index into memory as key-to-id entries at startup, then batch-loads
only the entity rows selected by a scan and caches them by id.
- **Run the same logic on the backend.** Because a table index is just a B-tree, the
same schema, selectors, and actions run against a persistent store on the
server (SQLite today, pg/mongodb in future). The runtime reads only the rows a
Expand Down Expand Up @@ -176,6 +179,20 @@ export async function createAppDB() {
`AsyncSqlDriver` is also exercised against Turso Database's browser WASM
engine in the shared driver conformance suite.

If all index keys fit in memory but all entity rows do not, use
`new SubscribableDB(new PreloadedHybridDB(primary))` instead. Its `loadTables`
call automatically preloads every index on every loaded table; scans resolve
ordered IDs in memory, including non-ID `uniqhash` value-to-ID pointers, and
batch-fetch only unresolved rows through the built-in `byId` entity index. No
explicit `preloadTables` call or extra B-tree `byIds` index is needed.
Repeated `loadTables` calls are incremental: previously loaded tables remain
available while supplied table definitions are added or refreshed.
Exact `byId` misses reconcile rows added by another connected runtime.
Apply changesets already persisted by another runtime sharing the primary with
`externalStorageMergeTrait`. Their normal merge operations update the preloaded
snapshot and invalidate subscribers, while external inserts persist
idempotently instead of failing as duplicates.

If your whole app state can be loaded into memory at startup, you may not need
`HybridDB`. A plain `new SubscribableDB(new DB(new BptreeInmemDriver()))` keeps
reads and writes synchronous, so you can use `useSyncSelector`, `useSyncDispatch`,
Expand Down Expand Up @@ -256,15 +273,15 @@ internally, so the same async subscription behavior is available without React.

## Entry points

| Import path | Contents |
| ---------------------------------------- | ------------------------------------------------------------------------------------ |
| `@will-be-done/hyperdb` | Core: `defineTable`, `v`, `selectFrom`, builders, `DB`, `HybridDB`, `SubscribableDB` |
| `@will-be-done/hyperdb/react` | React hooks and `DBProvider` |
| `@will-be-done/hyperdb/tracing` | Tracing store and tracer configuration |
| `@will-be-done/hyperdb/drivers/inmemory` | `BptreeInmemDriver` |
| `@will-be-done/hyperdb/drivers/sqlite` | `SqlDriver`, `AsyncSqlDriver` |
| `@will-be-done/hyperdb/drivers/idb` | `openIndexedDBDriver`, `IdbDriver` |
| `@will-be-done/hyperdb-devtool/react` | `HyperDBDevtools`, `HyperDBDevtoolsPanel` (separate package) |
| Import path | Contents |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `@will-be-done/hyperdb` | Core: `defineTable`, `v`, `selectFrom`, builders, `DB`, `HybridDB`, `PreloadedHybridDB`, `SubscribableDB` |
| `@will-be-done/hyperdb/react` | React hooks and `DBProvider` |
| `@will-be-done/hyperdb/tracing` | Tracing store and tracer configuration |
| `@will-be-done/hyperdb/drivers/inmemory` | `BptreeInmemDriver` |
| `@will-be-done/hyperdb/drivers/sqlite` | `SqlDriver`, `AsyncSqlDriver` |
| `@will-be-done/hyperdb/drivers/idb` | `openIndexedDBDriver`, `IdbDriver` |
| `@will-be-done/hyperdb-devtool/react` | `HyperDBDevtools`, `HyperDBDevtoolsPanel` (separate package) |

## Learn more

Expand Down
87 changes: 80 additions & 7 deletions packages/hyperdb-doc/src/content/docs/runtime/db.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: The DB Runtime
description: DB, SubscribableDB, HybridDB, transactions, lifecycle hooks, and traits.
description: DB, SubscribableDB, HybridDB, PreloadedHybridDB, transactions, lifecycle hooks, and traits.
sidebar:
order: 1
---
Expand All @@ -10,12 +10,13 @@ commands.

## Which runtime should I use?

| Runtime shape | Use when | Tradeoff |
| ------------------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DB` | You only need `selectSync`, `insert`, `upsert`, `deleteRows`, and transactions. | Lowest overhead. No subscriptions, reactive selector cache, revisions, or lifecycle hooks. |
| `SubscribableDB` + sync driver | Your reactive app can load its working state into memory. | Best interactive path: selectors and actions can stay synchronous with `useSyncSelector`, `useSyncDispatch`, `selectSync`, and `syncDispatch`. |
| `SubscribableDB` + async driver | Your reactive app should keep memory low and read directly from IndexedDB or async SQLite. | Uses async selectors/actions. Simpler than `HybridDB`, but every read follows the async driver path. |
| `SubscribableDB` + `HybridDB` | Local-first browser apps that want persistent storage plus fast reads for hot data. | Uses async APIs, but reads check the in-memory cache first. Missing index ranges fall through to the primary store, then get cached for next time. Writes update the cache first for immediate UI response, then flush to the primary store. |
| Runtime shape | Use when | Tradeoff |
| -------------------------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DB` | You only need `selectSync`, `insert`, `upsert`, `deleteRows`, and transactions. | Lowest overhead. No subscriptions, reactive selector cache, revisions, or lifecycle hooks. |
| `SubscribableDB` + sync driver | Your reactive app can load its working state into memory. | Best interactive path: selectors and actions can stay synchronous with `useSyncSelector`, `useSyncDispatch`, `selectSync`, and `syncDispatch`. |
| `SubscribableDB` + async driver | Your reactive app should keep memory low and read directly from IndexedDB or async SQLite. | Uses async selectors/actions. Simpler than `HybridDB`, but every read follows the async driver path. |
| `SubscribableDB` + `HybridDB` | Local-first browser apps that want persistent storage plus fast reads for hot data. | Uses async APIs, but reads check the in-memory cache first. Missing index ranges fall through to the primary store, then get cached for next time. Writes update the cache first for immediate UI response, then flush to the primary store. |
| `SubscribableDB` + `PreloadedHybridDB` | Index keys fit in memory, but retaining every entity row would be too expensive. | `loadTables` eagerly reads all tables once to build ID-only indexes. Scans never need persistent index work, but the first access to an entity row is asynchronous. Writes are persisted before the in-memory indexes are published. |

## `DB`

Expand Down Expand Up @@ -236,6 +237,78 @@ when a delete did not know the old row. That lets exact unique reads return from
memory while broader uncached scans still wait when a pending write could affect
their interval.

## `PreloadedHybridDB`

`PreloadedHybridDB` is the middle ground between range-cached `HybridDB` and a
fully resident `BptreeInmemDriver`. On `loadTables`, it preloads every declared
index on every loaded table. Each B-tree or hash leaf stores only the entity ID;
the startup rows used to build those keys are then released.

```ts
import {
DB,
PreloadedHybridDB,
SubscribableDB,
externalStorageMergeTrait,
execAsync,
} from "@will-be-done/hyperdb";
import { openIndexedDBDriver } from "@will-be-done/hyperdb/drivers/idb";

const primary = new DB(await openIndexedDBDriver("my-app"));
const db = new SubscribableDB(new PreloadedHybridDB(primary));

// Automatically preloads all indexes on both tables, including byId.
await execAsync(db.loadTables([tasksTable, projectsTable]));
```

`loadTables` is incremental. A later call keeps previously loaded tables and
adds or refreshes only the table definitions passed to that call.

A scan first reads its bounds from the in-memory ID-only index. This includes
non-ID `uniqhash` indexes, which are preloaded as unique value-to-ID pointers.
The built-in `byId` `uniqhash` is the canonical entity store: it begins with
unresolved ID entries, loads missing rows from the primary in batches, and
replaces those entries with hydrated rows. The result is returned in the order
produced by the scanned index. Repeating the scan—or reaching the same entities
through another index—reuses the hydrated `byId` entries.

An exact `byId` miss checks the primary before returning no row. This lets an
independently connected runtime, such as another browser tab, add a row after
the preload snapshot; discovering that row also adds its secondary index
pointers locally.

When merging a changeset that another runtime has already committed to the same
primary storage, add `externalStorageMergeTrait` alongside the trait that
suppresses local change tracking:

```ts
await asyncDispatch(
db.withTraits({ type: "skip-sync" }, externalStorageMergeTrait),
mergeChanges(args),
);
```

In this mode the merge compares against the preloaded snapshot. A row absent
from that snapshot follows the normal insert path, while persistence uses an
idempotent upsert because the external writer may already have stored it. The
committed insert/upsert/delete operations therefore update every preloaded
index and notify `SubscribableDB` subscribers normally. Use this trait only for
already-persisted external changesets; ordinary inserts retain duplicate-ID
checking.

`preloadTables` is unnecessary in this mode because `loadTables` always covers
all indexes, and tables do not need an extra B-tree full-scan index. Inserts,
upserts, deletes, and transactions update both the durable primary and the
ID-only indexes. Unlike `HybridDB`'s optimistic write path, these writes wait for
the primary operation before publishing the new in-memory index state.
Transactional hash-index changes use copy-on-write buckets and are published or
discarded together with the transaction.

Use this runtime when index keys are substantially smaller than complete rows
and startup can afford one bulk read of each table. Use regular `HybridDB` when
startup should touch only queried ranges, or a plain in-memory driver when all
rows comfortably fit in memory and reads must remain synchronous.

## Executing commands

Selectors and actions are generators. The dispatch and select helpers run them
Expand Down
16 changes: 11 additions & 5 deletions packages/hyperdb-doc/src/content/docs/runtime/drivers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ sidebar:
A driver is the actual storage backend behind a `DB`. The same selectors and
actions run unchanged against any driver, and in any environment. You can use a
single driver directly, or combine a persistent primary driver with an in-memory
cache through [`HybridDB`](/runtime/db/#hybriddb). You also choose whether to use
the sync or async runtime helpers, which depends on the storage path.
cache through [`HybridDB`](/runtime/db/#hybriddb), or preload ID-only indexes
through [`PreloadedHybridDB`](/runtime/db/#preloadedhybriddb). You also choose
whether to use the sync or async runtime helpers, which depends on the storage
path.

## Choosing a driver

Expand All @@ -21,16 +23,20 @@ the sync or async runtime helpers, which depends on the storage path.
| `AsyncSqlDriver` | `.../drivers/sqlite` | async | both | Async SQLite, including Turso WASM as a `HybridDB` primary |

Sync drivers work with `execSync` / `syncDispatch` / `selectSync`. Async drivers
require `execAsync` / `asyncDispatch` / `selectAsync`. `HybridDB` also uses the
async helpers, because a read may miss the memory cache and fall through to the
primary store.
require `execAsync` / `asyncDispatch` / `selectAsync`. `HybridDB` and
`PreloadedHybridDB` also use the async helpers, because a read may need entity
rows from the primary store.

A typical local-first browser setup uses `HybridDB` with IndexedDB or async
SQLite as the primary store and `BptreeInmemDriver` as the cache. If your whole
working set can be loaded eagerly, a plain `SubscribableDB` over
`BptreeInmemDriver` keeps the UI path fully synchronous. On the server, use a
native `SqlDriver` while running the _same_ schema, selectors, and actions.

All built-in drivers support the bulk table read used by
`PreloadedHybridDB.loadTables`. This path does not require a user-declared
full-scan index.

## In-memory

The simplest driver: a set of in-memory B+trees. Construct it with no arguments.
Expand Down
19 changes: 11 additions & 8 deletions packages/hyperdb-doc/src/content/docs/start/introduction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ both the client and server.
cache. Reads use cached index ranges when possible and fall through to the
primary store only for missing ranges. Writes update the cache first for
immediate UI feedback, then flush to the primary store.
- Index-preloaded reads: `PreloadedHybridDB` keeps every declared index in
memory with ID-only leaves, then batch-loads and caches entity rows only when
scans select them.
- JavaScript selectors and actions: selectors and actions are ordinary JS, with loops,
conditionals, and function calls. HyperDB gives you fast indexed lookups and
inserts underneath, not a query language to learn, and the same mental model on
Expand Down Expand Up @@ -93,14 +96,14 @@ npm install react react-dom

The core package ships several entry points:

| Import path | Contents |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `@will-be-done/hyperdb` | Core: `defineTable`, `v`, `selectFrom`, builders, `DB`, `HybridDB`, `SubscribableDB`, runtime helpers |
| `@will-be-done/hyperdb/react` | React hooks and `DBProvider` |
| `@will-be-done/hyperdb/tracing` | Tracing store and tracer configuration |
| `@will-be-done/hyperdb/drivers/inmemory` | `BptreeInmemDriver` |
| `@will-be-done/hyperdb/drivers/sqlite` | `SqlDriver`, `AsyncSqlDriver` |
| `@will-be-done/hyperdb/drivers/idb` | `openIndexedDBDriver`, `IdbDriver` |
| Import path | Contents |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `@will-be-done/hyperdb` | Core: `defineTable`, `v`, `selectFrom`, builders, `DB`, `HybridDB`, `PreloadedHybridDB`, `SubscribableDB`, runtime helpers |
| `@will-be-done/hyperdb/react` | React hooks and `DBProvider` |
| `@will-be-done/hyperdb/tracing` | Tracing store and tracer configuration |
| `@will-be-done/hyperdb/drivers/inmemory` | `BptreeInmemDriver` |
| `@will-be-done/hyperdb/drivers/sqlite` | `SqlDriver`, `AsyncSqlDriver` |
| `@will-be-done/hyperdb/drivers/idb` | `openIndexedDBDriver`, `IdbDriver` |

The React devtool ships as a separate package, `@will-be-done/hyperdb-devtool`,
exposing `HyperDBDevtools` from `@will-be-done/hyperdb-devtool/react`.
Expand Down
Loading
Loading