[improvement](hive) Push partition filters to HMS

[zhaorongsheng]

What problem does this PR solve?

Issue Number: close #67724

Related PR: #67739

Problem Summary: Hive partition pruning previously built the generic partition view by enumerating every HMS partition name before applying selective predicates. For highly partitioned tables, planning therefore depended on a full metastore listing. This change introduces a connector-filtered partition view for plain Hive tables, translates safe equality and IN predicates into the connector filter grammar during logical partition pruning, and builds the selected-partition map from HMS get_partitions_by_filter results. The existing full-list local pruning path remains the fallback when the HMS API or filter dialect is unavailable. A separate deferred partition state preserves batch split generation for no-filter full scans.

Known follow-up: logical pruning and physical handle preparation can still repeat connector-native partition filtering. The explicit result/handle propagation work is tracked in #67739.

Release note

Improve planning latency for selective Hive partition queries when HMS supports get_partitions_by_filter.

Check List (For Author)

• Test: Unit Test
  • HiveConnectorMetadataPartitionPruningTest (16 tests)
  • PluginDrivenExternalTablePartitionTest, PluginDrivenScanNodeBatchModeTest, PluginDrivenScanNodePartitionPruningTest, and PluginDrivenScanNodePartitionCountTest (41 tests)
  • DISABLE_BUILD_UI=ON ./build.sh --fe
  • git diff --check
• Behavior changed: Yes (selective Hive partition predicates are materialized through HMS before generic partition pruning while no-filter scans retain batch eligibility)
• Does this need documentation: No

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[924060929]

/review

[924060929]

Thanks for working on this. Pushing selective Hive partition predicates to HMS is the right direction, but I found one merge blocker and a few framework-boundary issues that should be clarified before merging.

P1: the current head does not compile.

In PruneFileScanPartition, nameToPartitionItem is reassigned on the connector-filtered and fallback paths, then captured by the lambda at line 147:

.or(() -> Optional.ofNullable(SortedPartitionRanges.build(nameToPartitionItem)));

Java only allows a lambda to capture a final or effectively-final local variable. A focused FE build on f5633377a1e11420ef6a101e5af2f63e2bb9aac9 fails with:

PruneFileScanPartition.java:[147,79]
local variables referenced from a lambda expression must be final or effectively final

The build reached fe-core after the preceding reactor modules compiled, so this is not the generated parser/proto mismatch mentioned in the PR description. Please fix this and rerun the FE compilation plus the two new fe-core tests. The 16 HiveConnectorMetadataPartitionPruningTest cases passed locally, but the fe-core tests could not start because main compilation failed.

The deferred partition state should be represented explicitly.

NOT_PRUNED and DEFERRED_PARTITION_PRUNING currently have identical field values and are distinguished only by singleton identity via ==. This creates a hidden invariant across logical rewrites, plan copies and physical translation. Please use an explicit state/enum, or another value-based representation, instead of object identity.

Relatedly, Math.max(nameToPartitionItem.size(), 1) stores a synthetic value in totalPartitionNum to distinguish a genuine prune-to-zero result from an unmaterialized partition universe. That field is also used for EXPLAIN partition=N/M and partition accounting, so a filtered table can be reported as 3/3 or 0/1 even when the real table has many more partitions. Please represent unknown total count/materialization state separately rather than encoding control state in a fake partition count.

There is also duplicated HMS work.

The logical pruning path calls listPartitions(filter) and obtains filtered HmsPartitionInfo, converts it to generic PartitionItem, and discards the connector-native metadata. Later PluginDrivenScanNode.convertPredicate() invokes Hive applyFilter() with the original predicate, which calls get_partitions_by_filter again to rebuild the HiveTableHandle. Thus one selective query can issue the same HMS filter RPC twice and still retain the full original predicate on BE because partial residual matching is not implemented.

I do not think this PR needs to redesign the entire external-table predicate/residual framework. The broader work to unify partition pruning, updated connector handles, and per-conjunct residual tracking can be a maintainer follow-up. However, the compile failure, identity-only deferred state, and synthetic total count are introduced by this change and should be addressed here. If the duplicate HMS RPC is intentionally left as a follow-up, please document it and add a tracking issue, ideally with evidence that the selective path still materially improves planning latency.

[924060929]

A suggested long-term architecture, for clarity only — I do not think this full refactor should be required in this PR:

The generic planner should model connector partition pruning as an explicit result-producing operation rather than requiring every engine to materialize the same eager Map<String, PartitionItem> lifecycle.

LogicalFilter + LogicalFileScan
        |
        | partition-relevant conjuncts, snapshot and scan parameters
        v
Connector partition-pruning interface
        |
        |-- Hive: HMS get_partitions_by_filter, with full-list fallback
        |-- Iceberg/Paimon: manifest or SDK expression pruning
        |-- Hudi: timeline and partition-path pruning
        |-- MaxCompute: remote partition-spec pruning
        v
PartitionPruningResult
        - explicit state: deferred / materialized / unsupported
        - selected partition domain or lazy partition source
        - updated ConnectorTableHandle / connector-native metadata
        - consumed conjunct indices and remaining predicate
        - exact vs approximate/superset result
        - OptionalLong totalPartitionCount, where unknown stays unknown
        v
PhysicalPlanTranslator -> PluginDrivenScanNode

The important properties would be:

1. LogicalFileScan should not eagerly enumerate every partition before the filter is available. The pruning rule should invoke the connector with only the partition-relevant conjuncts.
2. Each connector may use its native pruning model. A connector that does not have stable Hive-style partition names should not be forced through a Hive-shaped map merely to participate.
3. The result should carry the updated handle or connector-native partition metadata into the physical scan, so the scan does not repeat the same metastore/SDK pruning call.
4. Exact consumed conjuncts may be removed from the upper scan predicate; unsupported or approximate pushdown must remain as residual predicates for BE evaluation.
5. A remote failure or unsupported dialect should return an explicit unsupported/fallback result, not be inferred from an empty map or singleton identity.
6. Unknown total partition count should remain unknown. State and control flow should not be encoded in totalPartitionNum.
7. Full scans that need lazy or batched split generation should use an explicit lazy partition source/state rather than being repaired in doFinalize() after carrying an empty sentinel through logical planning.

This would support different external engines without adding engine-specific branches to Nereids or coupling logical pruning state to PluginDrivenScanNode finalization. For the current PR, a smaller safe implementation is reasonable; the full result type, handle propagation and residual unification can be tracked as maintainer-owned follow-up work.

[zhaorongsheng]

Thanks for the detailed review. Addressed in the latest head:

• Fixed the lambda capture compile failure in PruneFileScanPartition; the FE build now succeeds with UI disabled.
• Replaced identity-only deferred handling with an explicit SelectedPartitions state and represent unknown total partition count as unknown in EXPLAIN rather than a synthetic count.
• Preserved no-filter Hive full-scan batch eligibility by materializing the deferred partition state before the batch-mode gate.
• Added regression coverage for deferred materialization, batch eligibility, unknown totals, and zero-prune semantics. The relevant FE tests pass (41 tests), and the Hive metadata pruning test passes (16 tests).

The duplicate connector partition-pruning RPC is intentionally left out of this focused fix and is tracked in #67739.

[github-actions]

Requesting changes for eight substantiated issues found on the fixed PR head. The central selective-partition goal is not achieved on the production Hive binding path, several early/parallel consumers mis-handle the new deferred state, and the connector API/dialect boundaries are incomplete.

Checkpoint conclusions:

• Goal and test proof: not satisfied. Production binding enumerates the full partition view before the deferred initializer, and common integral predicates fall back to full listing. The added tests are fake/helper-level and miss these production paths; the existing SPI surface test is statically guaranteed to fail.
• Scope and focus: the diff is thematically focused, and there was no additional user-provided focus. The full 17-file change was reviewed.
• Concurrency and lifecycle: physical finalization materializes no-filter deferred scans before batch selection/dispatch, so that path is ordered correctly. However, the preload path now leaves full Hive partition materialization and the live filtered RPC under internal-table read locks, and async-MV collection consumes the deferred-empty map too early.
• Configuration: no new setting is introduced, but enable_preload_external_metadata no longer warms this partition view before locks.
• Compatibility: the public ConnectorCapability addition lacks the mandatory frozen-surface baseline and API-major bump. Hive 1/2/3 database addressing and Hive partition-name escaping were checked and matched their established paths.
• Parallel and special paths: synchronous and partition-batch planning, zero matches, non-partition predicates, unsupported filter fallback, metadata add/delete, SQL block rules, and no-filter scans were traced. The original residual preserves row filtering on stable metadata; the accepted mixed-generation range and predicate-flag defects remain.
• Coverage and results: no builds or tests were run, as required by the review environment. Changed tests do not cover production MVCC binding, no-filter MV rewrite, real typed HMS parsing, cache-generation mixing, SQL block rules, preload lock scope, or the SPI version gate.
• Observability: unknown totals render safely as ?, and HMS fallback is logged. One accepted path nevertheless marks a known full view unknown.
• Persistence and FE/BE propagation: no transaction, EditLog, failover, storage-write, or BE protocol change is involved; the state/filter changes are FE-only and are propagated through physical translation.
• Performance: the production pre-bind list, table-wide sorted-range cache, integral-literal fallback, and non-partition-filter full-view rebuild each undermine the stated optimization.
• Residual review status: all candidates were accepted, merged, or dismissed with evidence. After Round 1 found the issues below, all normal and risk-focused Round 2 reviewers returned NO_NEW_VALUABLE_FINDINGS. The review is converged on this exact head.

[github-actions]

[P1] Update the connector-plugin surface version with this enum. ConnectorPluginSurfaceTest freezes every ConnectorCapability constant, but connector-plugin-surface.txt has no entry for this value and both the POM and the pinned assertion still say 7.0. The existing test therefore fails, and more importantly an older major-7 FE can admit a newly built major-7 Hive plugin before linkage reaches this missing field. Please regenerate the baseline, bump connector.plugin.api.version to 8.0, and update the pinned assertion in this change.

[github-actions]

[P1] Do not encode an unmaterialized full scan as an empty used-partition set. A no-filter Hive LogicalFileScan never reaches PruneFileScanPartition, so QueryPartitionCollector reads this sentinel's empty map before scan finalization and records that the query uses zero partitions. PartitionCompensator treats that differently from ALL_PARTITIONS, and the async-MV path then excludes/rejects otherwise eligible partitioned MVs. Please teach the logical MV collector to handle DEFERRED explicitly (or materialize it before collection) and cover an unfiltered Hive-base-table MV rewrite.

[github-actions]

[P1] Defer the production Hive snapshot, not only this later logical initializer. Plain Hive tables are instantiated as PluginDrivenMvccExternalTable because Hive declares SUPPORTS_MVCC_SNAPSHOT. BindRelation calls StatementContext.loadSnapshots before constructing LogicalFileScan, and the ordinary latest path reaches materializeLatest -> listLatestPartitions -> metadata.listPartitions(..., Optional.empty()), building the full partition map before this method returns DEFERRED. Cold queries therefore still enumerate every HMS partition (and warm ones still rebuild/carry the full per-statement map). Please make the plain-Hive snapshot path lightweight/deferred too and test the real binding subclass.

[github-actions]

[P1] Build sorted ranges from the connector-filtered map. This branch gets partitionItems from a live HMS-filter result, but getSortedPartitionRanges(scan) uses PluginDrivenMvccExternalTable's separately cached full snapshot. After an external add, binary search can omit a matching key from the logical selected-name set used by partition-batch scans; after a delete, a stale matching range can reach the missing-item check below and abort planning. It also reintroduces the full-map work this path is meant to avoid. When connectorFilteredPartitions is true, bypass the table-wide cache and build ranges from exactly partitionItems (or carry a matching generation token).

[github-actions]

[P1] Preserve that the connector applied a partition predicate. HMS has already narrowed partitionItems before PartitionPruner runs; reapplying the same equality therefore retains the entire narrowed map, making result.hasPartitionPredicate false. That false value reaches SqlBlockRuleMgr, so require_partition_filter rejects a valid WHERE partition_col = ... query. Please combine the connector-filter fact with the local result (or otherwise retain the original-universe semantics) and add a rule-to-SQL-block test.

[github-actions]

[P1] Preserve literal type when rendering the HMS filter. In Hive 2.3's filter grammar, a quoted token is parsed as String, and MetaStoreDirectSql rejects it against an integral partition column; the JDO integral fallback is disabled by default. Thus common INT year/month predicates first issue a failing filtered RPC and then enumerate all partition names, defeating this feature. Render integral/date grammar forms when supported (and skip the HMS attempt for unsupported types), with a real typed-client test rather than the current STRING-only fake.

[github-actions]

[P2] Only mark/use this path when the connector actually applied a partition filter. A predicate such as data_col = 1 converts successfully, but Hive extracts no partition predicate; because the Optional filter is still present, listPartitions bypasses partitionViewCache and rebuilds the complete ConnectorPartitionInfo/PartitionItem view in O(all partitions). The lower name cache may avoid an HMS names RPC, but this line still labels the full result connector-filtered and reports an unknown total. Propagate an applied/not-applied result (or pre-check partition slots) so non-partition filters keep the cached full-view path.

[github-actions]

[P1] Keep Hive partition materialization outside internal table locks. Production Hive leaves supportsLatestSnapshotPreload false, so the pre-lock preload phase relies on initSelectedPartitions; it now receives only DEFERRED and does no partition warmup. Nereids then locks internal tables before BindRelation performs the full snapshot/list materialization and this rewrite performs the live get_partitions_by_filter RPC (or fallback). A mixed internal/Hive query therefore holds internal metadata locks across both operations. Please resolve/preload this view before locking, or use the preloaded full view for this lock-sensitive path.