Skip to content

Configurable RocksDB compression - #735

Open
cb1kenobi wants to merge 8 commits into
fix/prerelease-version-checkfrom
feat/compression-config
Open

Configurable RocksDB compression#735
cb1kenobi wants to merge 8 commits into
fix/prerelease-version-checkfrom
feat/compression-config

Conversation

@cb1kenobi

Copy link
Copy Markdown
Member

Summary

Exposes RocksDB block/blob compression as a per-column-family option (issue #201).

  • compression open option — an algorithm name ('lz4', 'zstd', 'zlib', 'none', …) or { algorithm, level }. Applies to both SST block compression and blob-file compression (large values ≥ 2 KB become blobs, whose compression otherwise defaults to none).
  • Default: LZ4 (applied natively in Database::Open) when the build supports it, else RocksDB's own default. Confirmed & documented that RocksDB's stock default is Snappy; we override to LZ4.
  • supportedCompression — module constant listing algorithms compiled into the build (from GetSupportedCompressions()). Opening with an unavailable algorithm throws.
  • db.compression getter — reads the live value via DB::GetOptions.
  • Native name↔enum mapping + supported list in the Node-free core/compression.{h,cpp} (GoogleTest-covered).

Build robustness (dynamic lib linking)

librocksdb.a references, but doesn't bundle, the compression libraries it was built with. binding.gyp now links exactly the compression static libs the resolved prebuild ships, enumerated at configure time by scripts/configure-rocksdb.mjs (which also provisions the pinned prebuild). This means the build works against both an older none+zlib prebuild and a compression-enabled one — no version-bump coupling. To actually activate LZ4, a compression-enabled prebuild must be pinned in package.json (currently 11.1.2); the feature is forward-compatible until then.

Where to look / open items for the reviewer

  • Cold open applies the requested compression to every existing column family, not just the one being opened (DBDescriptor::open). This is intended per the author, but worth a look — it means opening a multi-CF DB with a compression option changes all CFs' algorithm for subsequently-written files. A second in-process open that explicitly requests a different algorithm for an already-open CF now throws (DBRegistry::OpenDB); a plain reopen inherits the live setting (DBOptions::compressionExplicit gates this).
  • Block vs blob compression are unified (same algorithm for both). Splitting them (e.g. lz4 blocks + zstd blobs) was deliberately deferred (YAGNI); the object form extends cleanly later.
  • scripts/configure-rocksdb.mjs runs init-rocksdb at gyp configure time (before the prepare-rocksdb build action) so the enumerated lib set matches what will link. The Windows path (spawn + AdditionalDependencies splice) is unverified locally — worth watching on CI.

Testing

  • test/compression.test.ts — supported list, getter/defaults, validation, on-disk size comparison across multiple databases, already-open conflict cases, and a normalizeCompression guard suite.
  • test/native/compression_test.cc — name↔type mapping + supported-list GoogleTests.
  • benchmark/compression.bench.ts — RocksDB-only write/read throughput per available algorithm.

Notes

🤖 Generated with Claude Code

cb1kenobi and others added 7 commits July 29, 2026 15:50
Expose RocksDB block/blob compression as a per-column-family open option
(issue #201).

- `compression` open option accepts an algorithm name or `{ algorithm, level }`
  and applies to both SST block compression and blob-file compression
  (large values), which otherwise defaults to none.
- Default is LZ4 when the native build supports it, else RocksDB's own default
  (Snappy when linked, otherwise none). RocksDB's stock default is Snappy.
- `supportedCompression` module constant lists the algorithms compiled into the
  build (from rocksdb::GetSupportedCompressions()); build-dependent, so opening
  with an unavailable algorithm throws.
- `db.compression` getter reads the algorithm currently in effect via
  DB::GetOptions.
- Name<->enum mapping and the supported list live in the Node-free
  core/compression.{h,cpp} (GoogleTest-covered); applyCompression() in
  napi/helpers.cpp is shared by both CF-options builders.
- Vitest coverage: supported list, getter/defaults, validation, and on-disk
  size comparison across multiple databases with different settings.
- RocksDB-only benchmark comparing write/read throughput per algorithm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
librocksdb.a is a static archive that references, but does not bundle, whatever
compression libraries it was built with. Prebuilds vary: an older prebuild
ships only zlib, a compression-enabled one adds snappy/lz4/zstd/bzip2. Hardcoding
the full set breaks a clean build against an older prebuild ("no such file:
libsnappy.a"), while linking too few breaks against a compression-enabled one
("undefined symbols").

Enumerate the compression static libs actually present in deps/rocksdb/lib at
configure time (scripts/rocksdb-link-libs.mjs) and splice exactly those into the
link settings for both the binding and native-test targets, on POSIX and
Windows. librocksdb is still linked unconditionally.

The script reconciles deps/rocksdb to the pinned version first (init-rocksdb),
because gyp configure runs before the prepare-rocksdb build action, so the
installed prebuild may be absent or a different version than package.json pins —
either would otherwise yield a stale lib list baked into the link command.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Our RocksDB prebuilds are not built with Microsoft Xpress support (it is a
Windows-only, OS-native codec), so it never appears in supportedCompression and
listing it as a usable algorithm is misleading. The native name<->type mapping
still recognizes it, so a build that does enable it would still report it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold two single-purpose helpers back into their callers now that the
compression option has settled:

- applyCompression() had only two callers (createRocksDBColumnFamily and
  DBDescriptor::open); inline its block/blob-compression body at both and
  drop the declaration from helpers.h.
- Store.getCompression() was a one-line pass-through guarded by an open
  check the native getter already enforces; the db.compression getter now
  calls this.store.db.getCompression() directly.

No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the redundant prepare-rocksdb build action. rocksdb-link-libs.mjs
already downloads/installs the pinned prebuild at gyp configure time on
every reachable path (rebuild, native-test's node-gyp rebuild, and
build-only which errors without a prior configure), so the build-time
action only re-ran init-rocksdb as a no-op second time.

The link script now also fails the build when init-rocksdb can't launch
or exits non-zero — a non-zero exit aborts gyp configure — instead of
silently enumerating a stale/empty lib set that surfaces later as a
confusing link error. An empty enumeration remains valid (build-from-
source or a none-only prebuild ships no compression archives).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Reject a second in-process open of an already-open column family that
  explicitly requests a different compression algorithm (DBRegistry::OpenDB),
  instead of silently ignoring it. A plain reopen still inherits the live
  setting.
- Move the LZ4 default into the native layer (Database::Open) and stop the TS
  layer from injecting it; normalizeCompression now returns {} when unset. A new
  DBOptions::compressionExplicit flag keeps the default from tripping the
  already-open conflict check above.
- normalizeCompression now throws for a malformed object (missing algorithm),
  matching the string branch's unsupported-algorithm throw.
- Rename scripts/rocksdb-link-libs.mjs -> scripts/configure-rocksdb.mjs now that
  it also provisions the pinned prebuild, and refresh its header + binding.gyp
  reference.
- AGENTS.md: drop the stale applyCompression reference (it was inlined) and
  document the native default + compressionExplicit behavior.
- Tests: already-open conflict/same/plain-reopen cases and a normalizeCompression
  guard suite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cb1kenobi
cb1kenobi requested a review from kriszyp July 30, 2026 05:23

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces configurable block and blob compression (such as LZ4, Zstd, and Zlib) to rocksdb-js, including build-time library detection, native mapping logic, and public APIs to configure and query compression. The review feedback highlights several important issues: a missing optional flag when parsing the 'compression' option that would cause database opens to fail when omitted, an incorrect override of the default compression level to 0 when 'compressionLevel' is not specified, a typo in the Windows static library name (using 'zs.lib' instead of 'zlib.lib' or 'zlibstatic.lib'), and a recommendation to improve cross-platform script execution by spawning the Node executable with the tsx CLI entry point directly instead of relying on the shell wrapper on Windows.

Comment thread src/binding/database/database.cpp
Comment thread src/binding/database/database.cpp
Comment thread scripts/configure-rocksdb.mjs
Comment thread scripts/configure-rocksdb.mjs
configure-rocksdb.mjs launched provisioning through node_modules/.bin/tsx,
which on Windows is a .cmd/.ps1 wrapper requiring shell: true (slower, and
mis-parses a repo path with spaces). Invoke the current node executable against
tsx's CLI entry (tsx/dist/cli.mjs) directly — no shell, identical on every
platform. Addresses gemini-code-assist review feedback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cb1kenobi
cb1kenobi marked this pull request as ready for review July 30, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant