feat: slim down zvec dynamic libraries - #627
Open
chinaux wants to merge 10 commits into
Open
Conversation
chinaux
requested review from
Cuiyus,
egolearner,
feihongxu0824 and
zhourrr
as code owners
July 28, 2026 11:33
chinaux
force-pushed
the
feat/slim-c-api-library
branch
from
July 29, 2026 10:29
660ffdb to
b91f370
Compare
Reduce the FAT C API dylib size by ~37% through a 5-layer strategy with zero functional loss; core search code stays at -O3 for performance: - Compiler dead-code sectioning: -ffunction-sections -fdata-sections - Linker pruning: macOS -dead_strip + exported symbol list; Linux --gc-sections + version script, keeping only zvec_* C API symbols - Third-party feature trimming: Arrow disables Dataset & trims Compute kernels; RocksDB disables trace/iostats/perf contexts - Protobuf lite: proto uses LITE_RUNTIME, link libprotobuf-lite - Size-optimized third-party builds: -Os for RocksDB/glog/antlr4, MinSizeRel for Arrow
Extend the size-reduction work to the C++ all-in-one shared library and the Windows build: - libzvec C++ SDK (macOS/Linux): restrict exported symbols to zvec's own public API (namespace zvec, incl. nested zvec::ailego/core/turbo) via -exported_symbols_list / --version-script + -dead_strip / --gc-sections. Hides ~14.5k third-party symbols; libzvec.dylib 24.41MB -> 19.05MB (-21.9%), total -34.1% vs main baseline. Public C++ headers never expose third-party types, so external C++ consumers are unaffected. - Third-party libs (MSVC): favor small code (/O1 /Ob1) over speed (/O2) to mirror the -Os / MinSizeRel size optimization used on GCC/Clang, including Arrow's Release config.
…rom tests The slim compute patch removed kernels that production code paths rely on: - match_like (LIKE queries) from scalar_string_ascii/utf8 - sort_indices (ORDER BY) from vector_sort/vector_array_sort - make_struct / list_value_length from scalar_nested Restore these kernels in arrow.slim_compute.patch while keeping the rest (aggregates, temporal, round, random, etc.) removed. Arrow Dataset stays disabled: sql_expr_validator_test's dataset member was dead code and is removed; sql_expr_parser_test's scan test is rewritten with arrow::compute::ExecuteScalarExpression, preserving the original intent (verify a parsed expression evaluates correctly on real data) without depending on arrow::dataset. All 153 C++ tests pass locally (write_recovery_test flake passes in isolation).
…script The version script only exported zvec::* function/data symbols. vtable, typeinfo, VTT and guard variable symbols demangle to 'vtable for zvec::...' etc., which the zvec::* pattern does not match, so they were localized by 'local: *'. Consumers linking libzvec.so then failed with 'undefined reference to vtable for zvec::core_interface::HNSWIndexParam'. Add the mangled-name patterns (_ZTVN4zvec*, _ZTIN4zvec*, _ZTSN4zvec*, _ZTTN4zvec*, _ZGVN4zvec*), mirroring what exported_symbols_cpp.txt already does on macOS.
chinaux
force-pushed
the
feat/slim-c-api-library
branch
from
July 29, 2026 12:04
b91f370 to
3462df8
Compare
…ibility In linker version scripts, a quoted "zvec::*" entry is a literal name match (not a glob). GNU ld silently tolerates the never-matching entry, but lld (used by the Android NDK) enforces --no-undefined-version and fails with: ld.lld: error: version script assignment of 'global' to symbol 'zvec::*' failed: symbol not defined Keep only the unquoted glob pattern, which matches all demangled zvec:: symbols on both GNU ld and lld.
Introduces a self-contained implementation of the manifest on-disk format,
in preparation for dropping the libprotobuf/protoc dependency:
- pb_wire.h: minimal protobuf wire-format reader/writer (varint, fixed32/64,
length-delimited). Unknown fields are consumed and ignored, preserving
proto3 forward compatibility; malformed input (bad varint, out-of-range
length, groups, field number 0) is reported instead of crashing.
- manifest_enum.h: the on-disk enums, mirroring src/db/proto/zvec.proto with
identical numeric values. The CodeBooks in type_helper.h now speak these
instead of the generated protobuf enums.
- manifest_codec.{h,cc}: encodes/decodes all 16 messages directly between
zvec's C++ types and bytes, without an intermediate message object.
Format compatibility is established by two test suites:
- manifest_codec_test.cc cross-checks against libprotobuf while it is still
available: every message encoded by both implementations must produce
identical bytes, and each side must parse the other's output. This covers
the subtle cases - always-present base/quantizer_param sub-messages, the
absent quantizer_param of HNSW_RABITQ, Vamana's fixed32 alpha, empty
repeated string elements and oneof "last branch wins".
- manifest_codec_golden_test.cc pins protobuf's real output as embedded byte
arrays and does not depend on libprotobuf, so it keeps guarding the format
after the dependency is removed.
src/db/proto/zvec.proto is retained as the authoritative documentation of the
format. No behaviour change yet: Version::Save/Load still use protobuf.
Version::Load/Save no longer build an intermediate proto::Manifest; they read and write the manifest bytes with ManifestCodec instead. The on-disk format is unchanged - manifest_codec_test.cc verifies byte-for-byte equality with the protobuf implementation, and manifest_codec_golden_test.cc pins the format independently. Manifests are a few kilobytes, so the file is read/written in one go rather than streamed. ProtoConverter is intentionally kept for now: it is what the cross-check tests compare against. It is removed together with the protobuf dependency.
The manifest is now read and written entirely by ManifestCodec (added and cross-checked against protobuf in the previous commits), so libprotobuf and protoc are no longer needed. Removed: - thirdparty/protobuf submodule and its build integration - src/db/proto compilation (cc_proto_library, zvec.pb.cc, libprotobuf-lite link and zvec_proto target/deps across src and tests) - ProtoConverter and the cross-check test that depended on libprotobuf - the "build host protoc" stage from the Android and iOS CI workflows and the build_android.sh / build_ios.sh scripts, along with the now-unused GLOBAL_CC_PROTOBUF_PROTOC option. Cross-compiling no longer needs a host protoc, simplifying those pipelines noticeably. Kept: - src/db/proto/zvec.proto as the authoritative documentation of the on-disk manifest format - cc_proto_library in cmake/bazel.cmake as a generic helper (no longer used by zvec itself) The manifest on-disk format is unchanged; manifest_codec_golden_test.cc keeps guarding it. Full C++ test suite (156 tests) passes locally.
zvec.proto was retained as documentation when the protobuf dependency was
dropped, but the manifest format is now fully defined by manifest_codec.{h,cc}
(field numbers) and manifest_enum.h (enum values). Delete the file and the
src/db/proto directory, and repoint the doc comments that referenced it to the
codec instead.
Also drop the stale clang-tidy header cache entries for build/src/db/proto and
the src/db/proto/*.proto hashFiles input, which no longer exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR significantly reduces the size of zvec's prebuilt dynamic libraries with
zero functional loss, making the SDK far more practical to ship and embed.
Core search code stays at
-O3so query performance is unaffected.libzvec_c_api(C API, macOS)libzvec(C++ SDK, macOS)Motivation
Both dynamic libraries statically bundle heavy third-party dependencies (Arrow,
RocksDB, protobuf, antlr4, glog), and by default they export all of those
symbols and carry code paths zvec never uses. This inflates binary size, slows
down linking, and pollutes the global symbol namespace for downstream consumers.
Approach
1. Symbol visibility & dead-code elimination
-ffunction-sections -fdata-sectionsso unused code can be prunedat link time
-exported_symbols_list+-dead_strip--version-script+--gc-sectionslibzvec_c_apiexports onlyzvec_*C API symbolslibzvecexports only zvec's own public C++ API (namespacezvec, includingnested
zvec::ailego/zvec::core/zvec::turbo), hiding ~14.5kthird-party symbols
2. Third-party feature trimming
thirdparty/arrow/arrow.slim_compute.patch)3. Protobuf lite
src/db/proto/zvec.protoswitches toLITE_RUNTIMEand linkslibprotobuf-liteinstead of the full runtime.4. Size-optimized third-party builds
-Osfor RocksDB / glog / antlr4,MinSizeRelfor Arrow/O1 /Ob1(favor small code) instead of/O2, including Arrow'sRelease config — mirroring the
-Os/MinSizeRelbehavior on GCC/ClangCompatibility
src/include/zvec/keep theirexisting declarations; only visibility attributes and unused internals changed
as a patch during the build, keeping submodules pristine
Testing
macOS (arm64), Linux (x64/arm64) and Windows (x64)
nm/objdumpto confirm only theintended public symbols remain