Configurable RocksDB compression - #735
Conversation
Expose RocksDB block/blob compression as a per-column-family open option (issue #201). - `compression` open option accepts an algorithm name or `{ algorithm, level }` and applies to both SST block compression and blob-file compression (large values), which otherwise defaults to none. - Default is LZ4 when the native build supports it, else RocksDB's own default (Snappy when linked, otherwise none). RocksDB's stock default is Snappy. - `supportedCompression` module constant lists the algorithms compiled into the build (from rocksdb::GetSupportedCompressions()); build-dependent, so opening with an unavailable algorithm throws. - `db.compression` getter reads the algorithm currently in effect via DB::GetOptions. - Name<->enum mapping and the supported list live in the Node-free core/compression.{h,cpp} (GoogleTest-covered); applyCompression() in napi/helpers.cpp is shared by both CF-options builders. - Vitest coverage: supported list, getter/defaults, validation, and on-disk size comparison across multiple databases with different settings. - RocksDB-only benchmark comparing write/read throughput per algorithm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
librocksdb.a is a static archive that references, but does not bundle, whatever
compression libraries it was built with. Prebuilds vary: an older prebuild
ships only zlib, a compression-enabled one adds snappy/lz4/zstd/bzip2. Hardcoding
the full set breaks a clean build against an older prebuild ("no such file:
libsnappy.a"), while linking too few breaks against a compression-enabled one
("undefined symbols").
Enumerate the compression static libs actually present in deps/rocksdb/lib at
configure time (scripts/rocksdb-link-libs.mjs) and splice exactly those into the
link settings for both the binding and native-test targets, on POSIX and
Windows. librocksdb is still linked unconditionally.
The script reconciles deps/rocksdb to the pinned version first (init-rocksdb),
because gyp configure runs before the prepare-rocksdb build action, so the
installed prebuild may be absent or a different version than package.json pins —
either would otherwise yield a stale lib list baked into the link command.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Our RocksDB prebuilds are not built with Microsoft Xpress support (it is a Windows-only, OS-native codec), so it never appears in supportedCompression and listing it as a usable algorithm is misleading. The native name<->type mapping still recognizes it, so a build that does enable it would still report it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold two single-purpose helpers back into their callers now that the compression option has settled: - applyCompression() had only two callers (createRocksDBColumnFamily and DBDescriptor::open); inline its block/blob-compression body at both and drop the declaration from helpers.h. - Store.getCompression() was a one-line pass-through guarded by an open check the native getter already enforces; the db.compression getter now calls this.store.db.getCompression() directly. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the redundant prepare-rocksdb build action. rocksdb-link-libs.mjs already downloads/installs the pinned prebuild at gyp configure time on every reachable path (rebuild, native-test's node-gyp rebuild, and build-only which errors without a prior configure), so the build-time action only re-ran init-rocksdb as a no-op second time. The link script now also fails the build when init-rocksdb can't launch or exits non-zero — a non-zero exit aborts gyp configure — instead of silently enumerating a stale/empty lib set that surfaces later as a confusing link error. An empty enumeration remains valid (build-from- source or a none-only prebuild ships no compression archives). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Reject a second in-process open of an already-open column family that
explicitly requests a different compression algorithm (DBRegistry::OpenDB),
instead of silently ignoring it. A plain reopen still inherits the live
setting.
- Move the LZ4 default into the native layer (Database::Open) and stop the TS
layer from injecting it; normalizeCompression now returns {} when unset. A new
DBOptions::compressionExplicit flag keeps the default from tripping the
already-open conflict check above.
- normalizeCompression now throws for a malformed object (missing algorithm),
matching the string branch's unsupported-algorithm throw.
- Rename scripts/rocksdb-link-libs.mjs -> scripts/configure-rocksdb.mjs now that
it also provisions the pinned prebuild, and refresh its header + binding.gyp
reference.
- AGENTS.md: drop the stale applyCompression reference (it was inlined) and
document the native default + compressionExplicit behavior.
- Tests: already-open conflict/same/plain-reopen cases and a normalizeCompression
guard suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces configurable block and blob compression (such as LZ4, Zstd, and Zlib) to rocksdb-js, including build-time library detection, native mapping logic, and public APIs to configure and query compression. The review feedback highlights several important issues: a missing optional flag when parsing the 'compression' option that would cause database opens to fail when omitted, an incorrect override of the default compression level to 0 when 'compressionLevel' is not specified, a typo in the Windows static library name (using 'zs.lib' instead of 'zlib.lib' or 'zlibstatic.lib'), and a recommendation to improve cross-platform script execution by spawning the Node executable with the tsx CLI entry point directly instead of relying on the shell wrapper on Windows.
configure-rocksdb.mjs launched provisioning through node_modules/.bin/tsx, which on Windows is a .cmd/.ps1 wrapper requiring shell: true (slower, and mis-parses a repo path with spaces). Invoke the current node executable against tsx's CLI entry (tsx/dist/cli.mjs) directly — no shell, identical on every platform. Addresses gemini-code-assist review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary
Exposes RocksDB block/blob compression as a per-column-family option (issue #201).
compressionopen option — an algorithm name ('lz4','zstd','zlib','none', …) or{ algorithm, level }. Applies to both SST block compression and blob-file compression (large values ≥ 2 KB become blobs, whose compression otherwise defaults to none).Database::Open) when the build supports it, else RocksDB's own default. Confirmed & documented that RocksDB's stock default is Snappy; we override to LZ4.supportedCompression— module constant listing algorithms compiled into the build (fromGetSupportedCompressions()). Opening with an unavailable algorithm throws.db.compressiongetter — reads the live value viaDB::GetOptions.core/compression.{h,cpp}(GoogleTest-covered).Build robustness (dynamic lib linking)
librocksdb.areferences, but doesn't bundle, the compression libraries it was built with.binding.gypnow links exactly the compression static libs the resolved prebuild ships, enumerated at configure time byscripts/configure-rocksdb.mjs(which also provisions the pinned prebuild). This means the build works against both an oldernone+zlibprebuild and a compression-enabled one — no version-bump coupling. To actually activate LZ4, a compression-enabled prebuild must be pinned inpackage.json(currently11.1.2); the feature is forward-compatible until then.Where to look / open items for the reviewer
compressionto every existing column family, not just the one being opened (DBDescriptor::open). This is intended per the author, but worth a look — it means opening a multi-CF DB with a compression option changes all CFs' algorithm for subsequently-written files. A second in-process open that explicitly requests a different algorithm for an already-open CF now throws (DBRegistry::OpenDB); a plain reopen inherits the live setting (DBOptions::compressionExplicitgates this).scripts/configure-rocksdb.mjsrunsinit-rocksdbat gyp configure time (before theprepare-rocksdbbuild action) so the enumerated lib set matches what will link. The Windows path (spawn +AdditionalDependenciessplice) is unverified locally — worth watching on CI.Testing
test/compression.test.ts— supported list, getter/defaults, validation, on-disk size comparison across multiple databases, already-open conflict cases, and anormalizeCompressionguard suite.test/native/compression_test.cc— name↔type mapping + supported-list GoogleTests.benchmark/compression.bench.ts— RocksDB-only write/read throughput per available algorithm.Notes
mainonce Fix init-rocksdb skipping updates for pinned prerelease/revision versions #734 merges. The bottom three commits belong to Fix init-rocksdb skipping updates for pinned prerelease/revision versions #734.README.md(this library's docs); no separate documentation-repo change needed.🤖 Generated with Claude Code