feat(cubejs): enforce partition pruning in queryRewrite for cubes over partitioned event tables - #84
Merged
Merged
Conversation
…r partitioned event tables Owner rule (cxs2 spec 107 follow-up, 2026-09-03): a cube over cst.semantic_events must never reach ClickHouse without a predicate on the partition time column. A row-type cube whose time dimension is a payload path (simple_stays.started = properties.started_at) made every dashboard query read the JSON column across all monthly partitions: 10 GiB / 18.6M rows for a one-week POI query, killed by the server memory guard whenever two ran together. queryRewrite now derives a window from the query's explicit date ranges (timeDimensions.dateRange, inDateRange filters) and adds an inDateRange filter on the cube's partition dimension, widened by a margin on both sides. Policy per source table (default: semantic_events -> timestamp, 31 days), overridable with CUBEJS_PARTITION_PRUNING and per cube with meta.partition_dimension. It reuses the existing cube-to-table map, runs for every role, and stays out of the way when the query already constrains the partition column or states no explicit range. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nv1ngm1j8mV8rWsumTbpx2
…re default Local dev on Apple silicon runs cubestore under amd64 emulation, where the cache driver's WebSocket dies at startup and every query fails with 'Cube Store connection is closed'. Honouring the standard Cube env var (default unchanged: cubestore) lets the dev stack run with the in-memory driver; production configs set nothing and keep cubestore. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nv1ngm1j8mV8rWsumTbpx2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owner rule (cxs2 spec 107 follow-up, 2026-09-03): a cube over
cst.semantic_eventsmust never reach ClickHouse without a predicate on the partition time column.Why.
simple_stays.startedisproperties.started_at(a JSON payload path). Every dashboard query filtered on it read the JSON column across all monthly partitions: 10.15 GiB / 18.6M rows / 6.2 s for a one-week POI ranking (ClickHousesystem.query_log), and the server memory guard killed one of any two running together — the "Cube query error: 400" panels in Tychi's dashboards.What.
queryRewritederives a window from the query's explicit date ranges (timeDimensions.dateRange,inDateRangefilters) and adds aninDateRangefilter on the cube's partition dimension, widened by a margin on both sides. Policy per source table (defaultsemantic_events → timestamp, 31 days), overridable withCUBEJS_PARTITION_PRUNINGand per cube withmeta.partition_dimension. Reuses the existing cube-to-table map; runs for every role; no-op when the query already constrains the partition column or states no explicit range (relative ranges are left to Cube).Measured (same rows, Cube's generated shape): with the window 3.92 GiB / 1.77 s; with the window and the model also projecting only typed columns (separate model change) 29.9 MiB / 71 ms.
Tests:
src/__tests__/partitionPruning.test.js(5) — the suite's 6 pre-existing failures atmainare unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_01Nv1ngm1j8mV8rWsumTbpx2