Skip to content

fix: preserve global limit for multi-partition fetch#23800

Draft
discord9 wants to merge 1 commit into
apache:mainfrom
discord9:fix/limit-pushdown-global-fetch
Draft

fix: preserve global limit for multi-partition fetch#23800
discord9 wants to merge 1 commit into
apache:mainfrom
discord9:fix/limit-pushdown-global-fetch

Conversation

@discord9

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

An operator-level fetch on a plan with multiple output partitions is applied to
each partition. It can reduce work, but it does not satisfy a global limit. The
LimitPushdown rule could conflate those scopes and remove the remaining global
enforcement, producing too many rows or composing global and local limits
incorrectly.

What changes are included in this PR?

  • Track whether a pushed requirement still needs global enforcement.
  • Preserve or tighten per-partition fetches as early-stop hints without treating
    them as global limits.
  • Materialize a CoalescePartitionsExec or SortPreservingMergeExec boundary
    before a multi-partition global requirement is considered satisfied.
  • Keep OFFSET and per-partition LocalLimitExec semantics separate from the
    global fetch.
  • Ignore no-op global limits without changing pending limit state.

Are these changes tested?

Yes. Physical optimizer regression tests cover:

  • fetch-capable and unfetchable multi-partition inputs;
  • LIMIT, OFFSET + LIMIT, and OFFSET-only plans;
  • an existing smaller per-partition fetch;
  • Union input fan-out;
  • global limits above multi-partition local limits;
  • no-op global limits; and
  • fetch absorption at HashJoin without leaking fetch to its inputs.

The complete physical_optimizer::limit_pushdown integration test module and
workspace clippy were run from the final PR head.

Are there any user-facing changes?

Yes. Queries using these physical plan shapes now preserve the requested global
LIMIT/OFFSET rather than returning too many rows or over-restricting results.
There are no public API changes.

Signed-off-by: discord9 <discord9@163.com>
@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate labels Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LimitPushdown can mistake per-partition fetch for a global limit

1 participant