feat: IO-level dynamic filter pushdown into the native Delta scan (delta-kernel-rs predicate reuse)

[schenksj]

Disclosure: this issue was drafted with the help of an AI assistant .

Follow-up to #4807 (runtime dynamic filter pushdown for native hash joins). Blocked on the native Delta scan contrib landing (tracking PR #4366).

The generic phase of #4807 applies the join's DynamicFilterPhysicalExpr as a row-level filter above the probe-side scan, which the Delta contrib's DeltaKernelScanExec inherits with no format-specific work. This issue covers the IO-level upgrade so the filter also prunes inside the delta-kernel-rs read.

Why Delta is well positioned

DeltaKernelScanExec (contrib, PR #4366) is a kernel-read operator like IcebergScanExec — not a ParquetSource — so it does not get DataFusion's optimizer-driven scan pushdown. However, the contrib already ships a Spark -> delta-kernel predicate converter (contrib/delta/native/src/predicate.rs) used for static data skipping. The IO-level dynamic filter upgrade can reuse it rather than writing a new conversion layer.

Proposed shape

1. Snapshot the populated DynamicFilterPhysicalExpr at first poll of the probe-side scan stream (join build completes before the probe stream is polled; may require deferring kernel-read construction into the stream, mirroring the Iceberg follow-up).
2. Convert the convertible conjuncts (min/max bounds always; InList when present) through the existing predicate.rs converter and attach to the kernel scan's physical predicate. The HashTableLookupExpr strategy is not convertible; keep bounds only (row-level wrapper from feat: Runtime dynamic filter pushdown for native hash joins (build side -> probe side) #4807 still applies the full filter).
3. Verify how much row-group pruning delta-kernel-rs's default-engine Parquet read actually performs with an executor-side predicate — this bounds the payoff and should be measured before/while implementing.
4. File-level pruning is out of scope: the file list is resolved driver-side against Delta log stats.

Decision input

dynamic_filter_rows_pruned / selectivity metrics from #4807 on Delta star joins.