Add SQLite-backed persistent cache support for Node.js - #1
Open
nbransby wants to merge 3 commits into
Open
Conversation
…qlite Firestore offline persistence was previously browser-only, since the persistent cache is built entirely on IndexedDB. This adds a minimal IndexedDB implementation backed by the built-in node:sqlite module (Node.js >= 22.5), letting the entire existing indexeddb_* persistence stack (schema v18, migrations, LRU GC, client-side indexing) run unchanged on Node. - New platform seam src/platform/indexed_db.ts (per-platform via the existing rollup alias): isIndexedDbAvailable/openIndexedDb/ deleteIndexedDb + keyRange* factories. The browser implementation is a passthrough to the real globals; behavior there is unchanged. - Node adapter in src/platform/node/indexeddb/: order-preserving binary key codec (IDB key ordering == SQLite BLOB memcmp, incl. UTF-16 string semantics, binary keys and nested arrays), node:v8 structured-clone value serialization, FIFO transaction scheduling with request-queue drain and auto-commit, stateful re-seek cursors, autoIncrement key generators and unique index enforcement. - node:sqlite is loaded lazily via process.getBuiltinModule; on Node without it, persistence reports unavailable and the SDK falls back to the memory cache with the existing warning. - New PersistentCacheSettings.directory option (Node.js only) selects where the .sqlite files live, defaulting to ./.firestore; clearIndexedDbPersistence deletes the files. - Multi-tab remains unsupported on Node (clearer UNIMPLEMENTED error). - The USE_MOCK_PERSISTENCE/indexeddbshim test harness keeps working: the Node seam prefers a global indexedDB when one is installed. Verified: simple_db.test.ts (27 tests, cursors/indexes/ranges/aborts), local_store_indexeddb.test.ts (21 tests incl. index auto-creation with binary keys), new key codec property tests, and a new e2e offline test (write offline -> restart -> read from cache -> clear). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GCwoo1meAkgcbiJKZUqyyp
Multi-client spec tests rely on cross-tab WebStorage signaling. Under the Node SQLite persistence adapter there is no window/localStorage (persistence on Node is single-client), so gate those tests on WebStorageSharedClientState.isAvailable() the same way the runtime does. They continue to run in browsers and under the indexeddbshim harness (which installs a FakeWindow). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GCwoo1meAkgcbiJKZUqyyp
isClientZombied() used `this.webStorage?.getItem(...) !== null`, which evaluates to true when webStorage is null (undefined !== null). On platforms without LocalStorage — i.e. Node.js — every client was considered zombied, so a second persistence instance would ignore a live client's primary lease and steal it instead of failing with the exclusive-access error. Treat "no WebStorage" as "no zombie markers"; stale clients still age out via their lease/metadata timestamps. Also switch the remaining bare IDBKeyRange uses in indexeddb_persistence.test.ts to the platform keyRange factories and replace literal control characters in the key codec test with unicode escapes. All 44 indexeddb_persistence.test.ts tests now pass against the Node SQLite adapter, including the multi-instance primary lease arbitration tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GCwoo1meAkgcbiJKZUqyyp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Discussion
This PR adds support for Firestore persistent cache in Node.js environments using the built-in
node:sqlitemodule (available in Node.js 22.5+).Overview
This implementation provides a minimal IndexedDB adapter backed by SQLite for Node.js, enabling persistent local cache functionality that was previously unavailable in node environments. The adapter:
node:sqlitemodule viaprocess.getBuiltinModule()for lazy loading.firestore/)Key Changes
New Files
Core SQLite Adapter:
packages/firestore/src/platform/node/indexeddb/sqlite_engine.ts- SQLite connection management, schema catalog, and transaction handlingpackages/firestore/src/platform/node/indexeddb/key_codec.ts- Order-preserving binary encoding/decoding of IndexedDB keyspackages/firestore/src/platform/node/indexeddb/database.ts- IDBDatabase implementationpackages/firestore/src/platform/node/indexeddb/transaction.ts- IDBTransaction implementation with FIFO schedulingpackages/firestore/src/platform/node/indexeddb/object_store.ts- IDBObjectStore implementationpackages/firestore/src/platform/node/indexeddb/index.ts- IDBIndex implementationpackages/firestore/src/platform/node/indexeddb/cursor.ts- IDBCursor implementationpackages/firestore/src/platform/node/indexeddb/key_range.ts- IDBKeyRange implementationpackages/firestore/src/platform/node/indexeddb/key_path.ts- Key path evaluationpackages/firestore/src/platform/node/indexeddb/value_serializer.ts- Value serialization/deserializationpackages/firestore/src/platform/node/indexeddb/events.ts- Event and request handlingPlatform Integration:
packages/firestore/src/platform/node/indexed_db.ts- Node.js platform seam for IndexedDBpackages/firestore/src/platform/browser/indexed_db.ts- Browser platform seam (passthrough to globals)packages/firestore/src/platform/indexed_db.ts- Platform-agnostic interfaceModified Files
packages/firestore/src/api/cache_config.ts- Added_directoryfield toPersistentLocalCacheImplfor storing cache directory on Node.jspackages/firestore/src/api/database.ts- UpdatedclearIndexedDbPersistence()to handle Node.js database file deletionpackages/firestore/src/local/simple_db.ts- Refactored to use platform seam instead of direct@firebase/utilimportspackages/firestore/src/local/indexeddb_*.ts- Updated imports to use new platform seam for key range helperspackages/firestore/rollup.shared.cjs- Addedfsandv8to Node.js dependencies.gitignore- Added.firestore/directory for test cache filesTests
packages/firestore/test/unit/local/node_idb_key_codec.test.ts- Comprehensive tests for key encoding/decoding with property-based testingpackages/firestore/test/unit/local/node_sqlite_persistence.test.ts- End-to-end smoke test for offline persistence across process restartsAPI Changes
Added optional
directoryfield toPersistentCacheSettingsinterface to allow customization of the cache directory on Node.js.Testing
https://claude.ai/code/session_01GCwoo1meAkgcbiJKZUqyyp