Dphyper cascades - #15
Closed
TiNnNnnn wants to merge 56 commits into
Closed
Conversation
added 30 commits
August 25, 2026 01:09
…riable enable_dsl_rules.
…h, instead of CPatternLeaf. 2. Use orginal column form base table instead of output expression during binding.
added 26 commits
August 25, 2026 01:09
Rebind Project targets, support chained projection alternatives, and preserve LeftJoin shapes when loaded DSL rules require them. Add constraint diagnostics plus unit and OFF/ON end-to-end coverage. Keep local docs ignored.
Preserve DSL-visible logical shapes, adapt null-rejected full joins, and prove key-domain coverage through projections and grouping. Extend end-to-end coverage and make the bundled ANTLR runtime configure with CMake 4 on macOS.
Import redacted WeTune workload assets, execute parameterized traces through ORCA, distinguish planner fallback from DSL misses, and fix constraint-guided filter matching, typed predicate remapping, and invalid lower-bound costs.
Make DSL alternatives safe to insert into Cascades after native transformations have already created equivalent memo shapes. Re-root instantiated targets without recursively copying memo-bound children, and classify targets identical to their source as duplicate outcomes instead of reinserting no-op alternatives. Add relational memo reachability checks to both immediate duplicate-group discovery and final rehash. Reject ancestor/descendant group merges that would create indirect dependency cycles, while preserving the colliding expression in its original group so unsafe merges cannot leave an empty group. Extend DSL tracing with a duplicate stage plus uncompacted per-rule counters for binding attempts, bound symbols, rejection stages, generated alternatives, and duplicate alternatives. Emit query-level memo summaries containing group, duplicate-group, and group-expression counts. Teach the WeTune trace comparator that a canonically equivalent duplicate satisfies an expected rewrite, while keeping summary records outside application-set comparison. Update public E2E expectations to distinguish duplicate normalization from inserted alternatives and add framework regressions for both behaviors. Validated Calcite 74, 85, and 271 without stack overflow, fallback, or assertion; all finish with Optimizer: pg_orca and observed DSL outcomes. Also validated 25/25 public DSL E2E cases, gpopt_dsl_test, 16 Python trace-framework tests, and git diff --check.
Track whether a group expression was produced by a DSL shell directly or through a later native xform by following the immutable origin-expression chain. Apply the ancestor/descendant duplicate-group safety checks only to transformations and rehash collisions carrying DSL provenance. Native-only origin chains now take the original ORCA MarkDuplicates and FRehash paths without the added reachability filtering. Add a low-level regression covering native-only, direct DSL, and native-after-DSL provenance chains. Revalidated Calcite 74, 85, and 271, all 25 public DSL E2E cases, and the DSL unit and trace-framework suites.
Build an immutable per-root trie over admitted DSL source templates and use it to select candidate rules before invoking the full matcher. Model Input as a whole-subtree wildcard, preserve source-file rule order, and retain conservative fallbacks at virtual representation boundaries. Compile stable Filter, Project, aggregate, join, and union physical prefixes, including normalized Filter chains and Project limit/aggregate views. Keep the existing matcher, constraint checker, instantiator, memo binding, and rule attribution as the final correctness path. Add coverage for deep prefix discrimination, wildcard matching, fallback retention, ordering, normalized Filter roots, and Project roots. Route every DSL xform shell through candidate lookup. On the 37-rule Calcite hotspot this reduces matcher attempts from 605 to 215 while preserving the memo and applied rules; all 25 public E2E cases remain unchanged.
Walk DSL source-prefix tries directly while extracting memo bindings, preserving opaque Input alternatives without recursive Cartesian expansion. Keep native xforms on the existing CBinding path. Add transient join-spine views for predicates and IN subqueries pushed below binary or n-ary joins. Support both Select+ANY and LeftSemiApplyIn representations, while only crossing the preserved side of outer joins. Cover nested Filter remapping, pre/post unnest routing, n-ary joins, and null-supplying-side rejection. Solidus 126 now triggers the proven rule 27 then rule 11 chain.
Introduce CDSLMatchView as the shared read-only adaptation layer for aggregate/HAVING, fused order/limit, dedup identity, hidden limit shells, subquery carrier rebasing, and safe join-spine routing.\n\nMigrate Filter, InSub, Exists, Agg, Proj, and Sort/Limit matching to the shared view interface while keeping symbol binding and semantic validation operator-specific. Replace the standalone join-spine router and add direct view-layer unit coverage.
- match decorrelated IN/EXISTS through semi-join and filter-carrier views - remap predicates and aggregate shell inputs while preserving output schemas - guard transient empty memo groups during reachability checks - add query/per-rule alternative budgets and report skipped/exhausted applications - use generic-plan corpus traces and cover the new paths in unit and end-to-end tests
- retain aggregate output identity and InSub carrier phase during target construction - make memo duplicate merging and IndexNL costing robust under added DSL alternatives - prove Reference coverage from complete filter constraints before join elimination - implement null-aware PG18 NOT IN nested-loop translation - add forced-xform, join-key, and NULL-semantics end-to-end coverage
Recognize index, index-only, bitmap, foreign, and dynamic table access operators when resolving DSL table ownership for Reference, TableEq, and semantic column checks. Keep referred-side coverage checks conservative so filtered access paths cannot weaken foreign-key rewrite safety.\n\nClassify budget-truncated Cascades traces as inconclusive instead of missing rewrites, including downstream rules made unreachable by an upstream search cutoff. Add framework regressions for direct and transitive budget limits.
Validate trace manifests against physical rule-file IDs before selecting cases, so mismatched rule libraries cannot produce false alignment gaps. Expose DSL on/off, native xform suppression, and query/per-rule alternative budgets in the corpus runner. Preserve these controls and cumulative memo/rule search metrics in reports. Classify statement timeouts separately from query errors and retain partial DSL events from aborted queries for later diagnosis. Cover rule-file validation, cumulative metric aggregation, and timeout classification with framework tests.
Make the source-template prefix index consult query-local DSL alternative budgets before candidate selection and memo binding enumeration. Once a rule consumes its configured budget, remove its terminal and any now-empty trie branch before building more bindings. Preserve one budget_skipped summary event per pruned rule so differential reports remain explicitly inconclusive rather than treating controlled search truncation as a matcher miss. Unlimited budgets retain the existing traversal behavior. On shopizer:67 this removes 10,493 redundant attempts (60,336 to 49,843); shopizer:68 drops from 6,117 to 3,371 attempts while retaining all reference rewrites. Unit, 28-case E2E, and public Calcite corpus tests pass.
Extend exact trie edges for Proj* and Agg source roots with the same canonical GbAgg gates already enforced by their matchers: Global stage and no minimal-grouping provenance. Local/intermediate split alternatives are now rejected before memo binding construction and full rule matching. Keep matcher semantics authoritative and add a direct prefix-index test proving Global admission and Local rejection. The high-load shopizer:67 replay remains aligned while binding attempts fall from 49,843 to 17,895; public Calcite remains 4/4 and the 28-case DSL E2E suite passes.
Mark maximal binary InnerJoin regions through one shared path for both query preprocessing and DSL-generated alternatives, so CBO rewrites that change a join graph can re-enter DPHyper. Track completed graphs per Memo equivalence group and suppress only exact duplicates. Compare atom identity, predicate expressions, and unordered hyperedge topology so different Filter/Project placements and Cartesian skeleton cuts retain independent enumeration spaces. Keep failed budget attempts reusable, preserve native fallback behavior, add applied/reused fingerprint tracing, and extend unit and DSL E2E coverage for reordered graphs, changed predicates, Cartesian cuts, and native join-enumerator suppression.
Build CD-C eligibility metadata for Inner, LeftOuter, LeftSemi, LeftAnti, and FullOuter join edges. Derive SES from predicate columns, expand TES through associative and asscom conflict rules, and use ORCA null-rejecting evaluation for conditional outer-join algebra. Add directional applicability checks for non-inner joins and project eligible edges into a read-only DPHyper graph. Keep null-aware NOT IN outside this algebra so it continues to use native fallback. Cover legal and illegal outer-join rotations, semi/anti directionality, TES-constrained subset discovery, and unchanged pure-inner behavior with graph unit tests and existing DPHyper DSL E2E cases. Complex join Memo materialization remains disabled until receiver edge provenance is preserved.
|
Seems you are using me but didn't get OPENAI_API_KEY seted in Variables/Secrets for this repo. you could follow readme for more information |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.