feat(native): Add customSerializedValue routing for binary connector deserialization - #155
Open
20001020ycx wants to merge 219 commits into
Open
feat(native): Add customSerializedValue routing for binary connector deserialization#15520001020ycx wants to merge 219 commits into
20001020ycx wants to merge 219 commits into
Conversation
…on_parallelism session property (prestodb#27603) Summary: When RPCNode is in the output stage of a query (e.g., SELECT fb_llm_inference(...)), force_single_node_output places it in a SINGLE fragment with 1 task. For queries where RPCNode is not in the output stage (e.g., inside INSERT or before a JOIN/aggregation), RPCNode may already run distributed. This diff adds a session property to explicitly control RPCNode parallelism using a ROUND_ROBIN exchange, following the same pattern as remote_function_fixed_parallelism_task_count for Python/Thrift UDFs. Changes: - AddExchanges.java: visitRPC() inserts a ROUND_ROBIN exchange below RPCNode when rpc_function_parallelism > 1 - SystemSessionProperties.java: Add rpc_function_parallelism integer session property (default 0 = default planning, task count determined by query structure) Usage: SET SESSION rpc_function_parallelism = 4; SELECT fb_llm_inference(...) FROM big_table; Plan with rpc_function_parallelism=4: Fragment 0 [SINGLE]: Output -> GATHER Fragment 1 [ROUND_ROBIN 4 tasks]: Project -> RPCNode -> ROUND_ROBIN exchange Fragment 2 [SOURCE]: TableScan (distributed reads) Default (0): No exchange inserted — RPCNode uses default planning. For output-stage queries this means single task; for other query shapes it may be distributed. Differential Revision: D101288404
…prestodb#27614) (prestodb#27614) Summary: Upgrades the bundled Apache Iceberg version in presto-trunk from 1.10.0 to 1.10.1. This is a prerequisite for the Iceberg V3 Java support feature stack which requires APIs introduced in 1.10.1 such as PUFFIN deletion vector helpers and the V3 schema evolution methods. Also extracts inline `Request` arguments at 12 sites in `presto-main/src/test/java/com/facebook/presto/server/TestServer.java` into local variables to work around a javac target-type inference ambiguity that surfaces after the dependency bump (transitive deps shift causes javac to fail unifying `T = QueryResults` vs `T = RuntimeException` when the `Request` is built inline inside `client.execute(...)`). This matches the pattern already used elsewhere in the same file (e.g. lines 179-182). Without this fix, `presto-main` test-compile fails on JDK 17 with: TestServer.java:[236,42] incompatible types: inference variable T has incompatible equality constraints java.lang.RuntimeException, com.facebook.presto.client.QueryResults,T == NO RELEASE NOTES == Part of the Iceberg V3 Java support split (was D98749429). Reviewed By: zzhao0 Differential Revision: D101602649
## Description Upgrade gcs version to 2.2.28 Upgrade google-oauth-client version to **1.34.1** to address CVE-2020-7692 and CVE-2021-22573 ## Motivation and Context <!---Why is this change required? What problem does it solve?--> <!---If it fixes an open issue, please link to the issue here.--> ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan ``` presto> show schemas from gcs; Schema -------------------- information_schema ischema ischema1 schema01 schema02 tpch_sf1 (6 rows) Query 20260114_063500_00007_6ja2r, FINISHED, 1 node Splits: 19 total, 19 done (100.00%) [Latency: client-side: 0:01, server-side: 0:01] [6 rows, 87B] [10 rows/s, 155B/s] presto> CREATE TABLE gcs.schema01.table1th (id int, name varchar) WITH (external_location = 'gs://gcs_test_nv/schema01/table1th/'); CREATE TABLE Query 20260114_063735_00009_6ja2r, FINISHED, 0 nodes Splits: 0 total, 0 done (0.00%) [Latency: client-side: 0:04, server-side: 0:03] [0 rows, 0B] [0 rows/s, 0B/s] presto> INSERT INTO gcs.schema01.table1th VALUES (1, 'AVS'), (2, 'VP'), (3, 'PN'); INSERT: 3 rows Query 20260114_063811_00010_6ja2r, FINISHED, 1 node Splits: 19 total, 19 done (100.00%) [Latency: client-side: 0:14, server-side: 0:14] [0 rows, 0B] [0 rows/s, 0B/s] presto> select * from gcs.schema01.table1th; id | name ----+------ 1 | AVS 2 | VP 3 | PN (3 rows) Query 20260114_063859_00011_6ja2r, FINISHED, 1 node Splits: 23 total, 23 done (100.00%) [Latency: client-side: 0:02, server-side: 0:02] [3 rows, 4.73KB] [10 rows/s, 2.47KB/s] ``` ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == Security Changes * Upgrade google-oauth-client version to 1.34.1 to address `CVE-2020-7692 <https://github.com/advisories/GHSA-f263-c949-w85g>`_ and `CVE-2021-22573 <https://github.com/advisories/GHSA-hw42-3568-wj87>`_. ```
…2.25.4 (prestodb#27583) ## Description <!---Describe your changes in detail--> Bumps org.apache.logging.log4j:log4j-core from 2.25.3 to 2.25.4. Additionally, added log4j-slf4j2-impl as a runtime dependency in the Druid module to ensure consistent Log4j versions across Presto. Reason: The transitive dependency from druid-processing was pulling in log4j-slf4j2-impl:2.22.1, which resulted in a version mismatch with the updated Log4j APIs (2.25.4). Dependency tree before: <img width="792" height="413" alt="Screenshot 2026-04-14 at 12 19 13 PM" src="https://github.com/user-attachments/assets/8604ccd4-64b8-4f2b-afa6-2462642c59df" /> Dependency tree after: <img width="792" height="413" alt="Screenshot 2026-04-14 at 12 18 30 PM" src="https://github.com/user-attachments/assets/b413f10a-e7e1-4899-b5fb-afbd41a13bd9" /> ## Motivation and Context <!---Why is this change required? What problem does it solve?--> <!---If it fixes an open issue, please link to the issue here.--> ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan <!---Please fill in how you tested your change--> Druid connector is tested locally and the test result is attatched below: <img width="1496" height="575" alt="Screenshot 2026-04-14 at 2 51 59 PM" src="https://github.com/user-attachments/assets/89623dfe-5c64-4923-a541-8a4ebca2f70f" /> ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == Security Changes * Upgrade org.apache.logging.log4j:log4j-core from 2.25.3 to 2.25.4 inorder to address `CVE-2026-34480 <https://nvd.nist.gov/vuln/detail/CVE-2026-34480>`_. ```
…MV optimizer (prestodb#27538) (prestodb#27538) Summary: The MV query rewriter only overrode visitSimpleGroupBy, so column references inside CUBE, ROLLUP, and GROUPING SETS were never rewritten from base table columns to MV columns. Fix by adding visitCube, visitRollup, and visitGroupingSets overrides, and extending removeGroupingElementPrefix to handle all GroupingElement types. Differential Revision: D99539143 ``` == RELEASE NOTES == General Changes * Fix materialized view query rewriting for ``CUBE``, ``ROLLUP``, and ``GROUPING SETS`` clauses. Column references inside these grouping elements are now correctly rewritten to materialized view columns. ```
…todb#27595) ## Description Add operational JMX metrics to the sidecar plugin. ## Motivation and Context Operational metrics are important to understand how the system is behaving and to troubleshoot issues. ## Impact JMX metrics exposed. ## Test Plan CI ``` == NO RELEASE NOTE == ```
…st (prestodb#27519) ## Description Currently, as several major Iceberg test classes (TestIcebergDistributedQueries, IcebergDistributedSmokeTestBase, IcebergDistributedTestBase) continue to grow, the number of history queries tracked within the same `QueryRunner` for each test class has been increasing. This leads to growing memory usage that often causes OOM failures in CI testing. For example, after running all tests in `TestIcebergDistributedQueries`, the total number of history queries tracked in the queryRunner reaches 2900+, occupying approximately 600MB of additional memory. This PR explicitly sets `query.max-history` and `query.max-age` for the `QueryRunner` built in these test classes to significantly reduce this memory overhead. For example, after this change, after running all tests in `TestIcebergDistributedQueries`, the total number of history queries tracked in the queryRunner is around 200. ## Motivation and Context Reduce OOM during Iceberg CI tests. ## Impact N/A ## Test Plan N/A ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Tests: - Configure Iceberg distributed test query runners with low query.max-age and query.max-history values to cap in-memory query history during tests.
…n /presto-tests (prestodb#27606) Bumps [org.bouncycastle:bcprov-jdk18on](https://github.com/bcgit/bc-java) from 1.81 to 1.84. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/bcgit/bc-java/blob/main/docs/releasenotes.html">org.bouncycastle:bcprov-jdk18on's changelog</a>.</em></p> <blockquote> <!-- raw HTML omitted --> <!-- raw HTML omitted --> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/bcgit/bc-java/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/prestodb/presto/network/alerts). </details> ``` == RELEASE NOTES == Security Changes * Upgrade org.bouncycastle:bcprov-jdk18on from 1.81 to 1.84 to resolve `CVE-2026-0636 <https://nvd.nist.gov/vuln/detail/CVE-2026-0636>`_. ``` Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…prestodb#27632) Summary: Some queries have really large numbers reported per operator and these overflow during summation, which leads to the absence of plan in the query logging. To fix that we add functionality to cap the mentrics to still be able to see more or less reasonable numbers. ``` == NO RELEASE NOTE == ``` Differential Revision: D101851654
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
4 tasks
20001020ycx
force-pushed
the
yscope/feat/custom-serialized-value-routing
branch
3 times, most recently
from
April 22, 2026 18:28
b642d95 to
83f9113
Compare
) ## Description Add SQL filter pushdown for the Lance connector. Converts Presto's `TupleDomain` predicates to Lance SQL WHERE clause strings and pushes them down to the Lance scanner via `ScanOptions.Builder.filter()`. Lance evaluates filters natively during scan using its DataFusion-based SQL parser, reducing data read from disk. ### Key components - **`LanceSqlFilterBuilder`**: Converts `TupleDomain<ColumnHandle>` to SQL string. Supports Boolean, Integer, Bigint, Real, Double, Varchar, Date, Timestamp types with equality, IN, range, and IS NULL predicates. Column names are backtick-quoted for safety. - **Filter projection columns**: Columns needed for filter evaluation but not in query output are added to the Lance scan projection, ensuring Lance can evaluate the filter without scanning all columns. - **Safety**: The filter is returned as unenforced (following Iceberg's pattern), so Presto re-evaluates at executor level as a correctness guard. Unsupported types and complex filters (>100 ranges) are gracefully skipped — Presto handles them. ### Supported pushed-down predicates | Predicate | Example | |---|---| | Equality | `col = 42` | | Comparisons | `col > 30`, `col <= 100` | | IN lists | `col IN (1, 2, 3)` | | NULL checks | `col IS NULL` | | Range | `col >= 10 AND col < 20` | | Multi-column | predicates combined with `AND` | ### Supported types in filter literals | Presto Type | Lance Filter Literal | |---|---| | `BOOLEAN` | `true` / `false` | | `TINYINT/SMALLINT/INTEGER/BIGINT` | Integer literal | | `REAL` | Float literal | | `DOUBLE` | Double literal | | `VARCHAR` | `'string'` (single quotes escaped) | | `DATE` | `date '2024-01-15'` | | `TIMESTAMP` | `timestamp '2024-01-15 10:30:00.000000'` | No new dependencies — uses Lance's built-in SQL filter parser (DataFusion-based) via `ScanOptions.Builder.filter(String)`. ## Motivation and Context Without filter pushdown, Lance reads all rows from disk and Presto filters them at the executor level. With filter pushdown, Lance evaluates predicates natively during scan, significantly reducing I/O for selective queries. ## Impact New class and minor modifications in `presto-lance` connector only. No changes to existing Presto code. Documentation updated in `lance.rst`. ## Test Plan - All 49 presto-lance unit tests pass: `./mvnw test -pl presto-lance` - `TestLanceSqlFilterBuilder` — 17 new test cases covering: - Equality, IN lists, exclusive/inclusive ranges - Varchar with single-quote escaping - Date and timestamp literals - IS NULL, boolean, double - Multiple columns with AND - All-domain skip, multi-range OR disjunction - Nullable with values (OR IS NULL) - Existing tests updated for new constructor signature ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [x] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [x] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [x] Adequate tests were added if applicable. - [ ] CI passed. - [x] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes ``` == RELEASE NOTES == Lance Connector Changes * Add SQL filter pushdown to reduce data read from disk for selective queries. Supports equality, comparisons, IN lists, IS NULL, and range predicates on Boolean, Integer, Bigint, Real, Double, Varchar, Date, and Timestamp types. ``` --------- Co-authored-by: Steve Burnett <burnett@pobox.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…restodb#26959) ## Description Implements incremental refresh for materialized views. Instead of full recomputation, only stale partitions are refreshed using the IVM delta algebra established in DifferentialPlanRewriter. Iceberg integration is added in this PR. Depends on prestodb#26728 ## Motivation and Context Full MV refresh is expensive when only a subset of partitions changed. This solution only requires recomputing the updated partitions. ## Impact REFRESH MATERIALIZED VIEW automatically uses incremental refresh when possible, falls back to full refresh otherwise. ## Test Plan Extensive unit tests have been added. ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == General Changes * Add incremental refresh for materialized views Iceberg Connector Changes * Add incremental refresh for materialized views in the Iceberg connector ```
20001020ycx
force-pushed
the
yscope/feat/custom-serialized-value-routing
branch
2 times, most recently
from
April 22, 2026 20:01
bb977a3 to
b25d4db
Compare
Routes customSerializedValue in from_json() to ConnectorProtocol::deserialize() for the 5 handle types used by the CLP connector: ColumnHandle, ConnectorTableHandle, ConnectorTableLayoutHandle, ConnectorSplit, ConnectorTransactionHandle. Also adds serialize/deserialize virtuals to ConnectorProtocol.h for ColumnHandle, and VELOX_NYI stubs to presto_protocol_core.h.
20001020ycx
force-pushed
the
yscope/feat/custom-serialized-value-routing
branch
4 times, most recently
from
April 22, 2026 20:39
e1eea3d to
becf9fc
Compare
…-40490 (prestodb#27613) ## Description Upgrade async-http-client to 3.0.9 to address CVE-2026-40490 Dependency tree before fix : ``` [INFO] com.facebook.presto:presto-druid:presto-plugin:0.298-SNAPSHOT [INFO] +- org.apache.druid:druid-processing:jar:35.0.1:compile [INFO] | +- org.apache.commons:commons-compress:jar:1.27.1:compile [INFO] | +- commons-codec:commons-codec:jar:1.17.2:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile [INFO] | +- org.apache.commons:commons-text:jar:1.15.0:compile [INFO] | +- net.java.dev.jna:jna:jar:5.18.1:compile [INFO] | +- org.asynchttpclient:async-http-client:jar:3.0.2:compile [INFO] | | +- com.sun.activation:jakarta.activation:jar:2.0.1:compile [INFO] | | \- org.jetbrains:annotations:jar:26.0.2:compile [INFO] | +- org.hyperic:sigar:jar:1.6.5.132:compile [INFO] | \- com.github.oshi:oshi-core:jar:6.4.4:compile [INFO] | \- net.java.dev.jna:jna-platform:jar:5.13.0:compile [INFO] +- at.yawk.lz4:lz4-java:jar:1.10.2:runtime [INFO] +- com.facebook.airlift:bootstrap:jar:0.227:compile ``` Dependency tree after fix : ``` [INFO] com.facebook.presto:presto-druid:presto-plugin:0.298-SNAPSHOT [INFO] +- org.apache.druid:druid-processing:jar:35.0.1:compile [INFO] | +- org.apache.commons:commons-compress:jar:1.27.1:compile [INFO] | +- commons-codec:commons-codec:jar:1.17.2:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:compile [INFO] | +- org.apache.commons:commons-text:jar:1.15.0:compile [INFO] | +- net.java.dev.jna:jna:jar:5.18.1:compile [INFO] | +- org.asynchttpclient:async-http-client:jar:3.0.9:compile [INFO] | | +- com.sun.activation:jakarta.activation:jar:2.0.1:compile [INFO] | | \- org.jetbrains:annotations:jar:26.0.2:compile [INFO] | +- org.hyperic:sigar:jar:1.6.5.132:compile [INFO] | \- com.github.oshi:oshi-core:jar:6.4.4:compile [INFO] | \- net.java.dev.jna:jna-platform:jar:5.13.0:compile [INFO] +- at.yawk.lz4:lz4-java:jar:1.10.2:runtime [INFO] +- com.facebook.airlift:bootstrap:jar:0.227:compile ``` ## Motivation and Context <!---Why is this change required? What problem does it solve?--> <!---If it fixes an open issue, please link to the issue here.--> ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan Tested in local : <img width="1692" height="532" alt="image" src="https://github.com/user-attachments/assets/2c6c0715-4d89-442a-b6ae-5dec51dd5fbc" /> ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == Security Changes * Upgrade async-http-client to version 3.0.9 to address `CVE-2026-40490 <https://github.com/advisories/GHSA-cmxv-58fp-fm3g>`_. ```
…b#27625) ## Description Upgrade testing-library/react version to 16.3.2 ## Motivation and Context Using a more recent version helps avoid potential vulnerabilities and ensures we aren't relying on outdated or unsupported code. ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan <!---Please fill in how you tested your change--> ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Build: - Bump @testing-library/react dependency in presto-ui package.json from ^16.1.0 to ^16.3.2 and refresh yarn.lock accordingly.
…restodb#27480) (prestodb#27480) Summary: Converts the split_part_reverse implementation from a C++ Velox UDF to a Presto SQL-invoked scalar function (`SqlInvokedScalarFunction`), following the pattern in `ArraySqlFunctions.java` per reviewer feedback. The function body is a single SQL expression: ```sql RETURN ELEMENT_AT(SPLIT(str, delimiter), idx) ``` This leverages Presto's native `element_at()` negative index support on arrays: - Positive indices count from start (1-based), matching split_part - Negative indices count from end (-1 = last, -2 = second-to-last) - Returns NULL if |index| exceeds the number of parts - Index 0 throws an error (Presto native behavior) **Why SQL inline over C++ UDF:** - No Velox C++ compilation required - Globally available in all Presto queries (DaiQuery, pipelines, Bento) - One-liner SQL expression, trivially maintainable - Follows established pattern (ArraySqlFunctions, StringSqlFunctions) - Already registered via `SqlInvokedFunctionsPlugin` (StringSqlFunctions.class) **Usage:** ```sql SELECT split_part_reverse('foo/bar/baz/qux', '/', -1); -- 'qux' SELECT split_part_reverse('foo/bar/baz/qux', '/', -2); -- 'baz' SELECT split_part_reverse('foo/bar/baz/qux', '/', 1); -- 'foo' ``` This addresses T248997604 and incorporates feedback from the presto.dev Workplace thread. Differential Revision: D89498172
20001020ycx
force-pushed
the
yscope/feat/custom-serialized-value-routing
branch
from
April 24, 2026 15:06
becf9fc to
6f307dc
Compare
…odb#27624) ## Description Upgrade testing-library/jest-dom version to 6.9.1 ## Motivation and Context Using a more recent version helps avoid potential vulnerabilities and ensures we aren't relying on outdated or unsupported code. ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan <!---Please fill in how you tested your change--> ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Build: - Update presto-ui package.json and lockfile to use @testing-library/jest-dom version 6.9.1.
…stodb#27647) ## Description Upgrade react-data-table-component version to 7.7.1 ## Motivation and Context Using a more recent version helps avoid potential vulnerabilities and ensures we aren't relying on outdated or unsupported code. ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan <!---Please fill in how you tested your change--> ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Build: - Bump react-data-table-component in presto-ui package.json from 7.6.2 to 7.7.1 and refresh lockfile accordingly.
…tex to prevent pool name collision (prestodb#27900)
…E-2026-45205 (prestodb#27862) ## Description Upgrade commons-configuration2 to 2.15.1 to address CVE-2026-45205 ## Motivation and Context <!---Why is this change required? What problem does it solve?--> <!---If it fixes an open issue, please link to the issue here.--> Using a more recent version helps avoid potential vulnerabilities and ensures we aren't relying on outdated or unsupported code. ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan <!---Please fill in how you tested your change--> <img width="1048" height="809" alt="Screenshot 2026-05-25 at 9 32 23 PM" src="https://github.com/user-attachments/assets/2038013c-5011-47fc-91f1-17eddde3d061" /> <img width="974" height="742" alt="Screenshot 2026-05-25 at 9 32 37 PM" src="https://github.com/user-attachments/assets/a5c518af-79ca-459d-8687-5ffa2d21982f" /> <img width="1631" height="576" alt="Screenshot 2026-05-25 at 9 32 49 PM" src="https://github.com/user-attachments/assets/0b056070-24ab-4db5-9080-f26319e8e255" /> ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. `== RELEASE NOTES ==` ``` Security Changes * Upgrade commons-configuration2 in response to `CVE-2026-45205 <https://github.com/advisories/GHSA-337m-mw94-2v6g>`_. ```
``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Chores: - Update the presto-native-execution Velox dependency to the latest upstream commit.
… property to control Velox materialized exchange (prestodb#27881) Summary: Adds a new session property `native_exchange_materialization_enabled` (default: false) that controls whether the Velox native worker uses MaterializedOutput/MaterializedExchange operators in the Presto on Spark native codepath. When set to false, the native process falls back to PartitionAndSerialize + ShuffleWrite. The session property is read in NativeExecutionProcess.updateWorkerProperties() and unconditionally propagated to the C++ system config `exchange.materialization.enabled` before the native process starts. The session value always overrides the static config in both directions. Changes: - SystemSessionProperties: Add NATIVE_EXCHANGE_MATERIALIZATION_ENABLED constant, boolean property registration (default false), and static getter - NativeExecutionSystemConfig: Add EXCHANGE_MATERIALIZATION_ENABLED constant and default - NativeExecutionProcess.updateWorkerProperties(): Set exchange.materialization.enabled from session property value - TestNativeExecutionSystemConfig: Add new property to all test maps Reviewed By: xiaoxmeng Differential Revision: D106582232 ``` == RELEASE NOTE == General Changes * Add `native_exchange_materialization_enabled` session property (Presto on Spark native codepath only) to control whether Velox native workers use MaterializedOutput/MaterializedExchange operators. When set to `true`, enables materialized exchange; when `false` (default), falls back to PartitionAndSerialize + ShuffleWrite. ```
Summary: Extends the AST-level MV query rewriter to substitute a base table inside a JOIN with a materialized view when the MV's columns fully cover that table's usage in the query. Uses the shared MaterializedViewExpressionRewriter (from the parent diff) for all expression-level rewriting. Components: - MaterializedViewJoinQueryRewriter: top-level driver for JOIN MV rewriting - JoinRewriteContext: handles join tree transformation via AstVisitor, safety validation, MV freshness checks - Recursive tryRewrite: after one leaf is swapped, recurses to try remaining leaves - collectJoinLeaves via DefaultTraversalVisitor: walks the join tree to find leaf tables Safety guards when MV has GROUP BY: - Reject if query has no GROUP BY (MV would collapse rows) - Reject aggregates over non-swapped table columns (fan-out changes results) - Reject LEFT/RIGHT JOIN on the preserved side, and FULL OUTER JOIN - COUNT(*) rewritten to SUM(mv.cnt) when MV has a matching column - Upfront isEligibleForRewrite validation (no exception-based rejection) ``` == RELEASE NOTES == General Changes * Add JOIN support to the materialized view query optimizer. Queries that join a base table covered by a materialized view with another table can now be rewritten to scan the materialized view in place of the base table, subject to safety guards (matching GROUP BY, no aggregates over non-swapped tables, supported join types). ``` Differential Revision: D103812393
Address modernizer plugin issues reported by our modernizer tool. This was debt from the JDK upgrade because we disabled the modernizer on all modules
…restodb#27699) Summary: Differential Revision: D103025710 ## Description Wires the driver-side metadata sidecar into the presto-on-spark bootstrap. After Airlift initializes, when both `built-in-sidecar-functions-enabled=true` and `sparkProcessType == DRIVER`, `PrestoSparkInjectorFactory` invokes `WorkerFunctionRegistryTool.getWorkerFunctions()` to fetch native function metadata from the sidecar and registers the result into `FunctionAndTypeManager` as built-in `SqlInvokedFunction`s. Also adds the default `OptionalBinder<SidecarBinaryLocator>` binding (returns `Optional.empty`) to `DriverSidecarModule` so deployments can override with their own concrete locator without requiring a binding from this module. ## Motivation and Context D103025711 introduced the metadata-sidecar machinery but did not call it. This diff adds the bootstrap call site, mirroring the equivalent code path in `PrestoServer` for Prestissimo's coordinator-side sidecar registration. The companion change in `PrestoFacebookSparkServiceFactory` (Meta-internal) installs `DriverSidecarModule` on driver and executor JVMs so Airlift's strict-config check accepts the `metadata-sidecar.*` properties on both tiers. ## Impact Disabled by default — `built-in-sidecar-functions-enabled` defaults to `false` in `FeaturesConfig`. When enabled by an operator, the only behavior change is a one-time bootstrap step that registers additional built-in functions; no effect on already-resolvable function references or query execution paths. ## Test Plan End-to-end validation in D103025708 (paste P2298835496) — boots the sidecar from a real native worker binary on the driver, registers `koski_cosine_similarity` (a native-only Velox function with no Java implementation), plans an INSERT that uses it, and confirms native execution writes the expected rows. ## Contributor checklist - [x] My PR adheres to the code style of this project. - [x] My code builds clean without any errors or warnings. - [x] I am willing to help maintain this change if there are any issues in the future. ## Release Notes ``` == RELEASE NOTES == General Presto-on-Spark Changes * Update the driver-side metadata sidecar registration of worker functions into the Airlift bootstrap ```
…spatch (prestodb#27908) ## Description - Widen the try-catch in `ExpressionOptimizer.cpp` to cover `velox::expression::optimize()` in addition to `tryEvaluateToConstant()` - Previously, exceptions thrown during expression optimization (e.g. Velox type dispatch failures on UNKNOWN type) escaped uncaught and crashed the sidecar process - Now any `VeloxException` or `std::exception` during optimization is caught and returned as a structured `NativeSidecarFailureInfo` error response ## Motivation and Context Resolves prestodb#27907. Uncovered by prestodb#27011. Velox's type dispatch macros (`VELOX_DYNAMIC_TEMPLATE_TYPE_DISPATCH` etc.) do not handle `TypeKind::UNKNOWN` and throw `VeloxRuntimeError` via `VELOX_FAIL`. This affects ~20 Presto functions (array_except, array_intersect, contains, array_distinct, etc.) when given empty array literals or NULL arrays. The existing try-catch only wrapped `tryEvaluateToConstant` but the crash occurs earlier in `optimize()` during function resolution. Moving the try-catch to cover the entire optimization pipeline prevents the sidecar from crashing. ## Impact This fix is narrow and impacts only the correctness of expressions optimized by the native expression optimizer. Other possible fixes were considered and dropped in favor of this: 1. Catch all `VeloxException`s including `VeloxRuntimeError`s in Velox expression optimizer: pramodsatya/velox@8e9f188 **Pros:** Fix in Velox itself, `makeFailExpr` produces proper failure expression that flows through normally, the sidecar's `toVeloxExpr` → `optimize` → `veloxToPresto` path works end-to-end, returns a structured failure with the error message **Cons:** Catches broader than intended in Velox (RuntimErrors during constant folding are usually bugs), requires Velox PR 2. Fix each Presto function in Velox to guard against `UNKNOWN` type **Where:** Each of the ~20 affected functions in `velox/functions/prestosql/` **What:** Either: - Switch to `_ALL` macros (requires `UnknownValue` to be hashable — not currently possible for set-based functions) - Add explicit `if (elementType->isUnknown()) { return special_impl; }` guards before dispatch (like `approx_distinct` and `merge` already do) **Pros:** Each function handles UNKNOWN correctly at its own level; could even produce correct results for trivial cases (empty arrays) **Cons:** 20+ function changes, each needs its own logic, large Velox PR, doesn't protect against future functions that forget the guard ## Test Plan e2e testcase added. ## Release Notes ``` == NO RELEASE NOTE == ```
``` == NO RELEASE NOTE == ```
prestodb#27668) Co-authored-by: Reetika Agrawal <reetika.agrawal@ibm.com>
…restodb#27498) ## Description Minor code refactoring of Sphinx config `presto-docs/src/main/sphinx/conf.py`: - The PR replaces all single quotes with double quotes for strings to ensure consistency. - The PR replaces the modulo operator (%) and replace method call with f-strings for text formatting. ## Motivation and Context Align Python code with the current standards. ## Impact No ## Test Plan Build the documentation and check that the pages are correct: ```shell presto-docs/build open target/html/router/deployment.html ``` <img width="1323" height="816" alt="image" src="https://github.com/user-attachments/assets/f274196d-fdd6-4dd7-804f-b6a1449f0783" /> ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Refactor the Sphinx documentation configuration for Presto to standardize string formatting and modernize text interpolation. Enhancements: - Standardize all string literals in the Sphinx config to use double quotes for consistency with code style guidelines. - Replace legacy percent and replace-based string formatting with f-strings in the Sphinx configuration to align with modern Python practices. Signed-off-by: Denis Krivenko <dnskrv88@gmail.com>
prestodb#27903) ## Summary `protocol::AggregationNode::aggregations` is `std::map<VRE, Aggregation>` on the native side (sorted by variable name); the Java side iterates `LinkedHashMap` insertion order. When the two orders diverge — e.g. `approx_distinct_*` mixed with `sum_*`, alphabetical sort `approx_distinct < sum` but Java inserts `sum`s first — the native `AggregationNode` output schema differs from the Java planner's. Channel positions shift and type mismatches surface at exchange operators: ``` type_->kindEquals(vector.type()) Type mismatch: BIGINT vs. DOUBLE Operator: LocalPartition(...) ``` prestodb#27493 partially addressed this by switching three Java optimizer rules from `HashMap` to `LinkedHashMap` (deterministic insertion order), but did not align Java with the native side's sort. Queries whose variable names happened to sort consistently with their insertion order passed; others still crashed. This adds an explicit `aggregationOutputs: List<VariableReferenceExpression>` field, populated on the Java side from `aggregations.keySet()` in `LinkedHashMap` order. The native converter uses this list when present and falls back to `std::map` iteration only for older coordinators that don't send the field (so rolling upgrades are safe). Fixes prestodb#27902. ## Changes - `presto-spi/.../AggregationNode.java`: `@JsonProperty(READ_ONLY) getAggregationOutputs()` — derived from `aggregations`, no constructor change. - `presto_protocol_core.h`/`.cpp`: `List<VRE> aggregationOutputs` on `protocol::AggregationNode`, with backward-compatible `from_json`. - `PrestoToVeloxQueryPlan.cpp`: converter uses `aggregationOutputs` when non-empty; falls back to map iteration otherwise. - Tests: protocol round-trip + backward-compat (`AggregationNodeTest.cpp`); Java getter ordering (`TestPreAggregateBeforeGroupId`). ## Test plan - [ ] `mvn test -pl presto-main-base -Dtest=TestPreAggregateBeforeGroupId#testAggregationOutputsPreservesInsertionOrder` - [ ] `presto-native-execution` build + run `presto_protocol_test --gtest_filter="AggregationNodeTest.*"` - [ ] Existing `TestPreAggregateBeforeGroupId` suite stays green - [ ] Manually verify a reproducer query (`approx_distinct(...)` + `sum(...)` under `GROUPING SETS` with `optimizer.pre_aggregate_before_grouping_sets=true`) no longer crashes on a Prestissimo cluster ## Release notes \`\`\` == RELEASE NOTES == Native Execution Changes * Fix runtime type-mismatch crashes at exchange operators in Prestissimo when aggregation variable names sort differently from their Java allocation order. The protocol now carries an explicit aggregation output ordering so native workers build the AggregationNode output schema in the order the Java planner intended. \`\`\` ## Summary by Sourcery Ensure native AggregationNode output schemas follow Java planner aggregation ordering by explicitly propagating aggregation output variables through the protocol and converter. Bug Fixes: - Prevent type-mismatch failures in native execution caused by differing aggregation iteration orders between Java and C++. Enhancements: - Expose an explicit aggregationOutputs list on AggregationNode in the Java SPI and carry it through the native protocol for deterministic aggregation output ordering. Tests: - Add Java and C++ regression tests to verify aggregation output ordering is preserved and that the protocol remains backward compatible when aggregationOutputs is absent.
…ic connector support (prestodb#27653) ## Description - Hoist `registerDynamicFunctions()` before `registerVeloxConnectors()` in `PrestoServer.cpp` - De-inline connector factory registration functions from `Registration.h` into `Registration.cpp` ## Motivation and Context This PR enables dynamic connector plugins — connectors loaded at runtime via `plugin-dir` using `dlopen()` — as described in [RFC-0019: Connector Plugins](https://github.com/prestodb/rfcs/blob/main/RFC-0019-connector-plugins.md). Two changes are needed to make this work correctly. Both changes revolve around a singleton factory map defined in `Registration.cpp`: ```cpp static std::unordered_map<std::string, std::shared_ptr<ConnectorFactory>> factories; ``` This map lives inside `detail::connectorFactories()` and maps connector names (e.g. `"hive"`, `"tpch"`) to their `ConnectorFactory` instances. Both `registerConnectorFactory(name, factory)` and `getConnectorFactory(name)` read and write this map. ### Why `registerDynamicFunctions()` must be hoisted before `registerVeloxConnectors()` `registerVeloxConnectors()` iterates over every `*.properties` file in the `catalog/` directory. For each one, it reads `connector.name` and calls `getConnectorFactory(connectorName)` to look up the factory in the singleton map, then calls `factory->newConnector(...)` to instantiate the connector. `registerDynamicFunctions()` loads plugin `.so` files from the `plugin/` directory via `dlopen()`. Each plugin's `registerExtensions()` entry point calls `registerConnectorFactory(...)` to insert its factory into the same singleton map. Before this PR, `registerVeloxConnectors()` ran before `registerDynamicFunctions()`. Since plugins had not been loaded yet, the factory map had no entry for dynamically registered connector names. When `registerVeloxConnectors()` processed a catalog .properties file referencing a dynamic connector, `getConnectorFactory()` threw because the corresponding factory had not been registered yet. So the sequence must be: 1. `registerDynamicFunctions()` — loads plugin `.so` files, plugins call `registerConnectorFactory()` to insert into the map 2. `registerVeloxConnectors()` — reads `catalog/*.properties`, calls `getConnectorFactory()` to look up entries in the map The original code had no problem with built-in connectors (Hive, TPC-H, etc.) because their factories are registered statically at compile time. ### Why registration functions must be de-inlined Before this PR, the factory map was a static local variable inside the inline function connectorFactories() in Registration.h. Any connector shared library that #includes this header gets its own copy of the static variable, meaning both presto_server and the plugin library hold separate factory maps (Note, shared library references the header file in the presto-native-execution for compilation, the implementation are resolved at run time): ``` presto_server → factory map at 0xAAAA {"hive": ..., "tpch": ...} libpresto_plugin.so → factory map at 0xBBBB {"my_connector": ...} ``` This breaks the singleton property of the factory map: the plugin's `registerConnectorFactory()` inserts into its own map (0xBBBB), but the server's `getConnectorFactory()` reads from its own map (0xAAAA) — and finds nothing. De-inlining connectorFactories() ensures the static local variable exists in exactly one place (Registration.cpp, linked into presto_server), allowing dynamically registered connector known to presto_server successfully. ## Test Plan - [x] Build `presto_server` with the changes - [x] End-to-end tested with a customized native connector shared library (.so) loaded via plugin-dir, verifying successful connector registration and query execution. ## Release Notes ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery Enable dynamic connector plugins to be correctly registered and discovered by Presto's native server. Enhancements: - Move connector factory registry implementation out of the header to ensure a single shared factory map across server and plugins. - Adjust server startup sequence to load dynamic functions before registering Velox connectors so plugin-based connectors can be instantiated.
…p-java to latest (prestodb#27922) ## Description - Upgrade dorny/paths-filter to v4.0.1 per https://github.com/dorny/paths-filter/releases. Fixes prestodb#27918 - Upgrade actions/checkout to v6.0 per https://github.com/actions/checkout/releases - Upgrade actions/set-java to v5.2 per https://github.com/actions/setup-java/releases ## Motivation and Context Node 20 will be deprecated 16-Jun-2026. ## Impact Now CI jobs will be able to use non-deprecated and futuristic Nodejs 24+ ## Test Plan Ensure CI is all green. ## Contributor checklist - [x] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [x] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [x] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == NO RELEASE NOTE == ``` ## Summary by Sourcery CI: - Bump dorny/paths-filter from v3 to v4.0.1 across multiple GitHub Actions workflows to keep CI up to date with supported Node versions.
…tries (prestodb#27015) ## Description <!---Describe your changes in detail--> Presto ST_Equals function for empty geometries should return true regardless of geometry types. ## Motivation and Context <!---Why is this change required? What problem does it solve?--> <!---If it fixes an open issue, please link to the issue here.--> Closes prestodb#26253 ## Impact <!---Describe any public API or user-facing feature change or any performance impact--> ## Test Plan <!---Please fill in how you tested your change--> Added in tests. ## Contributor checklist - [ ] Please make sure your submission complies with our [contributing guide](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md), in particular [code style](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#code-style) and [commit standards](https://github.com/prestodb/presto/blob/master/CONTRIBUTING.md#commit-standards). - [ ] PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced. - [ ] Documented new properties (with its default value), SQL syntax, functions, or other functionality. - [ ] If release notes are required, they follow the [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines). - [ ] Adequate tests were added if applicable. - [ ] CI passed. - [ ] If adding new dependencies, verified they have an [OpenSSF Scorecard](https://securityscorecards.dev/#the-checks) score of 5.0 or higher (or obtained explicit TSC approval for lower scores). ## Release Notes Please follow [release notes guidelines](https://github.com/prestodb/presto/wiki/Release-Notes-Guidelines) and fill in the release notes below. ``` == RELEASE NOTES == General Changes * Update ST_Equals function for empty geometries to return true regardless of geometry types. * Add config properties for legacy ST_Equals behavior. ``` ## Summary by Sourcery Adjust ST_Equals geometry comparison semantics for empty geometries of the same or different types. Bug Fixes: - Ensure ST_Equals returns true for empty geometries of the same type and false for empty geometries of different types. Tests: - Add ST_Equals relation tests covering empty geometries of matching and differing types.
…todb#25139) ## Description Addresses prestodb#25104 Currently, Presto does not support INSERT INTO operations on bucketed but unpartitioned Hive tables. This limitation originates from a hard check in HiveWriterFactory: https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/HiveWriterFactory.java#L480 ## Motivation and Context Supporting writes to bucketed unpartitioned Hive tables in Presto would improve compatibility and enhance Presto’s ability to handle modern Hive table layouts. It's a reasonable and useful feature for users who wish to leverage bucketing for performance optimizations even without partitioning. ## Impact This change would align Presto’s behavior with the broader SQL-on-Hadoop ecosystem and remove an artificial limitation that may block valid use cases — particularly in data warehousing environments where bucketing is used independently of partitioning. ## Release Notes ``` == RELEASE NOTES == Hive Connector Changes * Add support for INSERT into bucketed but unpartitioned Hive tables in Hive, including follow-up fixes for native validation and insert handling. ```
…tent with other handle types Move the customSerializedValue check after getSubclassKey() and add the VELOX_CHECK guard against $-prefixed internal types, matching the pattern used by all other handle types.
…ke const& Align with the other new serialize stubs (ColumnHandle, ConnectorPartitioningHandle, ConnectorIndexHandle) which all take const&. The non-const ref was inconsistent and semantically wrong for a serialize operation.
20001020ycx
force-pushed
the
yscope/feat/custom-serialized-value-routing
branch
from
June 5, 2026 02:58
097a555 to
0a7c63a
Compare
Use folly::Endian::big instead of htons/ntohs for byte-swapping, which is already a linked dependency and makes the big-endian intent explicit rather than relying on the network-order abstraction.
20001020ycx
force-pushed
the
yscope/feat/custom-serialized-value-routing
branch
from
June 5, 2026 03:02
0a7c63a to
1843179
Compare
…test CMake FOLLY_INCLUDE_DIRS was empty on CI, making target_include_directories a no-op. Folly::folly as a CMake imported target automatically provides both include directories and link libraries, matching how other targets in the project reference folly.
…dian.h
folly/Endian.h does not exist in this version of folly — the Endian
class is defined in folly/lang/Bits.h. Also revert Folly::folly back
to ${FOLLY_LIBRARIES} as the include path issue was a red herring.
…estodb#27499) ## Summary Add a new optimizer rule `PullConstantProjectionAboveExchange` that moves constant assignments from `ProjectNode`s below remote `ExchangeNode`s to a new `ProjectNode` above the exchange. This avoids serializing and shuffling constant values across the network. ### Before ``` Exchange(REMOTE) Project(a = col_a, c = CONSTANT) Source(col_a) ``` ### After ``` Project(a = a, c = CONSTANT) Exchange(REMOTE) Project(a = col_a) Source(col_a) ``` ## Rule behavior - Matches remote `ExchangeNode`s whose sources are all `ProjectNode`s - Identifies constant assignments that are identical across all sources (for multi-source/`UNION` exchanges) - Skips constants used in partitioning, ordering, or hash columns - Strips pulled constants from source projects and narrows the exchange output layout - Adds a new project above with identity mappings + pulled constants - Gated behind the `pull_constant_projection_above_exchange` session property (default: **disabled**) ## Loop prevention (gated) To prevent oscillation with `PushProjectionThroughExchange` (which would otherwise push the pulled-up constant projection straight back down), that rule skips pushing projections consisting solely of symbol references and constants. This skip is **gated on the same `pull_constant_projection_above_exchange` session property**, so when the feature is disabled the default plan is unchanged. ## Optimizer placement The rule runs as a standalone `IterativeOptimizer` step **after** the other optimizers, so that any optimizer that may generate constants below exchanges has had a chance to run first. ## Changes | File | Change | |------|--------| | `FeaturesConfig.java` | New `optimizer.pull-constant-projection-above-exchange` config property | | `SystemSessionProperties.java` | Expose as `pull_constant_projection_above_exchange` session property | | `PullConstantProjectionAboveExchange.java` | **New** — the optimizer rule | | `PushProjectionThroughExchange.java` | Gated loop-prevention: skip pushing symbol+constant projections when the feature is enabled | | `PlanOptimizers.java` | Register the rule as a standalone iterative step | | `properties-session.rst` | Document the new session property | | `TestPullConstantProjectionAboveExchange.java` | **New** — 11 unit tests | | `TestPushProjectionThroughExchangeConstants.java` | **New** — 4 interaction tests | | `TestFeaturesConfig.java` | Config property test coverage | | `TestLocalQueries.java` | End-to-end tests (enabled vs disabled) | ## Test Plan - 11 unit tests: single/multi source, same/different constants, partitioning-key exclusion, local-exchange skip, session-property gating, non-project source, all-constants guard, and a determinism test verifying non-deterministic expressions are **not** pulled. - 4 interaction tests for the push rule: identity+constant blocked (feature on), all-constants blocked (feature on), non-trivial expressions still pushed, identity-only blocked. - End-to-end `TestLocalQueries#testPullConstantProjectionAboveExchange` compares results with the optimization enabled vs disabled (basic constants, constant + join, multiple constants + filter, `UNION ALL`, and a non-deterministic expression). - `TestFeaturesConfig` covers the new property; existing `PushProjectionThroughExchange` behavior is unchanged when the feature is disabled. ## Release Notes ``` == RELEASE NOTES == General Changes * Add session property ``pull_constant_projection_above_exchange`` (disabled by default) that pulls constant projection assignments above remote exchanges, avoiding serialization and shuffling of constant values across the network. :pr:`27499` ``` ## Summary by Sourcery Introduce a planner optimization that pulls constant projection assignments above remote exchanges and wires it into the optimizer behind a configurable session property, including safeguards to avoid oscillation with existing projection pushdown. New Features: - Add a PullConstantProjectionAboveExchange rule that hoists constant projections above remote exchanges to avoid shuffling constant values. - Expose a new optimizer session property pull_constant_projection_above_exchange (and corresponding config flag) to toggle the constant-projection pull-up behavior. Enhancements: - Adjust PushProjectionThroughExchange to skip pushing projections made only of symbols and constants when constant-pull-up is enabled to prevent optimizer ping-pong. - Register the new rule as a late-stage iterative optimizer so it runs after other constant-generating optimizations. Documentation: - Document the pull_constant_projection_above_exchange session property in the admin session properties reference. Tests: - Add rule-level tests for PullConstantProjectionAboveExchange covering single/multi-source exchanges, partitioning key exclusion, locality, non-project sources, all-constant guards, non-deterministic expressions, and session gating. - Add interaction tests for PushProjectionThroughExchange to verify behavior with constant and non-trivial projections when the feature is enabled or disabled. - Extend TestLocalQueries to validate end-to-end correctness with the optimization enabled versus disabled. - Update FeaturesConfig tests to cover the new configuration property.
…odies (prestodb#27917) Summary: `MaterializedViewOptimizationRewrite` only overrode `visitQuery`, so MVQO ran on bare `SELECT` only. CTAS and INSERT that wrap a `Query` were left unchanged because the default `visitNode` doesn't recurse. Add `visitCreateTableAsSelect` and `visitInsert` overrides that recurse into the inner `Query` and rebuild the wrapping statement only when MVQO rewrote it. The `query_optimization_with_materialized_view_enabled` session-property gate is unchanged. Reviewed By: abhash09 Differential Revision: D107310587 ``` == RELEASE NOTES == General Changes * Add the materialized view query optimizer to the inner query of ``CREATE TABLE AS`` and ``INSERT`` statements, not just bare ``SELECT``. ```
…pe/feat/custom-serialized-value-routing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add
customSerializedValuerouting infrom_json()for all 10 connector handle types. WhencustomSerializedValueis present in the incoming JSON, the routing Base64-decodes it and dispatches toConnectorProtocol::deserialize()instead of the default JSON-to-struct parsing.Motivation and Context
Enables C++ plugin connectors to use binary serialization codecs instead of JSON, as proposed in prestodb/rfcs#49.
This is the C++ worker-side counterpart to the Java
ConnectorCodec/CodecSerializerpipeline (prestodb#26257).Test Plan
presto_serverwith the changesplugin-dirRelease Notes