branch-4.1: [fix](mv) Prevent snapshot reads from using current MVs #67700 - #67729
Open
github-actions[bot] wants to merge 1 commit into
Open
branch-4.1: [fix](mv) Prevent snapshot reads from using current MVs #67700#67729github-actions[bot] wants to merge 1 commit into
github-actions[bot] wants to merge 1 commit into
Conversation
## Problem An external-table query that explicitly reads an older snapshot can still enter materialized view rewrite. The candidate materialized view represents the table state at its refresh snapshot, so using a current materialized view for a historical query can silently return data from the wrong point in time. ## Root cause `LogicalFileScan` stores standard `FOR TIME AS OF` and `FOR VERSION AS OF` clauses in `tableSnapshot`. The materialized-view eligibility checker rejected scans with table samples or scan parameters, but did not inspect `tableSnapshot`. As a result, the historical scan was treated like an ordinary latest-snapshot scan. ## How to reproduce 1. Create an Iceberg table and insert an initial row, then record that snapshot ID. 2. Insert newer data and refresh a materialized view over the table at the current snapshot. 3. Enable materialized-view rewrite and query the Iceberg table with `FOR VERSION AS OF <old_snapshot_id>` (the same issue applies to `FOR TIME AS OF`). 4. Before this change, the historical query can be considered eligible for rewrite by the current-snapshot materialized view, producing current rather than historical results. The same condition can be reproduced directly in the optimizer by building a `LogicalFileScan` with a non-empty `tableSnapshot`: the table-query-operator checker previously returned false. ## Fix Treat a non-empty `LogicalFileScan.tableSnapshot` as a table-level query operator, alongside table samples and scan parameters. This conservatively prevents materialized-view rewrite until the optimizer can prove that the query snapshot and materialized-view refresh snapshot are semantically equivalent. Add a unit test that constructs a file scan with a version snapshot and verifies that the checker rejects it from ordinary rewrite eligibility.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #67700