storage, encoded ops: add dictionary encoding infrastructure + encoded operations (Phases 1-4)#1
Open
devin-ai-integration[bot] wants to merge 3 commits into
Conversation
Implements the foundation for SingleStore-style encoded columnstore operations in TiFlash: - Add CompressionCodecDictionary (0x96) for low-cardinality columns - Supports Int8/Int16/Int32/Int64 and String data types - Bit-packed dictionary IDs with configurable cardinality threshold (4096) - Automatic fallback to raw storage when cardinality exceeds threshold - Full compress/decompress roundtrip with header format - Add ColumnDictionary type for in-memory encoded representation - Holds dictionary entries + per-row packed IDs - Supports filter/permute directly on encoded data - decode() method for materialization when needed - Update CompressionMethodByte enum (Dictionary = 0x96) - Update CompressionMethod enum (Dictionary = 7) - Register codec in CompressionCodecFactory - Update CompressionSettings maps - Add DictEncodingMeta proto message in dmfile.proto - Tracks cardinality, bit_width, encoding status per column segment - Optional fields for backwards compatibility (proto2 semantics) - Add comprehensive unit tests for codec and ColumnDictionary Feature is gated behind tiflash_enable_dictionary_encoding (default: OFF). No dictionary segments are written unless explicitly enabled, ensuring full backwards compatibility with existing deployments. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
…decDictionary - Add DataTypes/IDataType.h include for DataTypePtr - Use move semantics for PaddedPODArray (constructor takes rvalue ref) - Implement explicit copy constructor for COWPtr clone() support - Implement all pure virtual methods from IColumn interface (insertRangeFrom, insertManyFrom, insertDisjunctFrom, insertManyDefaults, replicateRange, scatter, scatterTo, gather, getExtremes) - Fix -Wunused-parameter in decompressDataInteger/decompressDataString - Return MutableColumnPtr from cloneResized (matches base interface) All new object files compile cleanly with clang-14 -Werror. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Phase 2 — EncodedFilter: Pre-compute filter predicates on dictionary entries (N), apply per-row via ID lookup (M rows) — O(N + M) instead of O(N*M). Supports eq, ne, IN, LIKE predicates. Phase 3 — EncodedGroupBy: Array-indexed aggregation using dictionary IDs as direct array indices. Supports SUM, COUNT, MIN, MAX, ANY plus multi-aggregate in a single pass. Phase 4 — EncodedStarJoin: Fused scan→join→aggregate pipeline on dictionary IDs. Builds dimension hash table, pre-computes join for each dictionary entry, then single pass over fact IDs accumulating into group buckets. Also adds DictionaryEncodingEngine for encoding decisions and column encoding/decoding, CMakeLists integration, and comprehensive tests (48 test cases across 4 test files). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration
Bot
changed the title
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary: Implement SingleStore-style encoded columnstore operations in TiFlash for 5-30x query speedups on analytical queries with low-cardinality columns.
What is changed and how it works?
Phase 1 — Dictionary Encoding Infrastructure:
CompressionCodecDictionary(0x96 byte): bit-packed dictionary IDs for columns with ≤4096 distinct values, transparent fallback to raw storageColumnDictionarytype: holds dictionary entries + packed IDs, supports filter/permute/decode operations without full materializationPhase 2 — Encoded Filters (
EncodedFilter):eq,ne,IN,LIKEpredicatestryApply*()methods detectColumnDictionaryinput and use encoded fast pathPhase 3 — Encoded Group-By (
EncodedGroupBy):MAX_ENCODED_GROUPS = 4096Phase 4 — Encoded Star Joins (
EncodedStarJoin):Supporting:
DictionaryEncodingEngine: encoding decisions + column encoding/decodingEncoded/directoryCheck List
Tests
Side effects
Documentation
Release note
Link to Devin session: https://6sense.devinenterprise.com/sessions/3710f99abca4415ba59b856f0a5588fc
Requested by: @premal