fix(deps): update datafusion ecosystem

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
datafusion (source)	dependencies	major	50.3 → 53.0
deltalake	dependencies	minor	0.29.4 → 0.32.0
object_store	dependencies	minor	0.12 → 0.13

---

Release Notes

apache/datafusion (datafusion)

v53.1.0

Compare Source

v53.0.0

Compare Source

v52.5.0

Compare Source

v52.4.0

Compare Source

v52.3.0

Compare Source

v52.2.0

Compare Source

v52.1.0

Compare Source

v52.0.0

Compare Source

v51.0.0

Compare Source

delta-io/delta.rs (deltalake)

v0.32.3

Full Changelog

Merged pull requests:

• docs: update the changelog for 0.32.2 #​4459 (rtyler)
• feat: add support for variant type #​4325 (abhiaagarwal)

v0.32.2

Full Changelog

Closed issues:

• [Bug]: DynamoDB setup instruction in docs #​4452
• [Feature]: Support passing datatypes other than string in custom_metadata in CommitProperties from python #​4443
• [Bug]: can't change partition column by overwriting table #​4434
• [Bug]: Python 1.5.1 MERGE regression: small source batch uses substantially more RSS than 1.5.0 #​4422
• [Bug]: python import deltalake crashes on linux kerner with 64KB page size #​4413
• [Bug]: Clarification (or regression) dataSkippingNumIndexedCols #​4398

Merged pull requests:

• refactor: deprecate legacy DeltaScan codec surface #​4458 (ethan-tyler)
• docs: update dynamodb locking setup #​4456 (fizyxbt)
• fix: normalize commit metadata handling #​4455 (ethan-tyler)
• fix: restore merge safe cast behavior #​4454 (ethan-tyler)
• chore: adopt the released buoyant_kernel version #​4453 (rtyler)
• test(core): consolidate integration tests into two binaries #​4451 (rtyler)
• fix(core): add checked RecordBatchWriter constructor #​4450 (fksegundo)
• fix(python): gate jemallocator off on FreeBSD (fall back to system allocator) #​4449 (sahuagin)
• fix: support Windows UNC paths #​4447 (rtyler)
• chore: clippy and dependency API fixes #​4446 (rtyler)
• chore: upgrade to the buoyant_kernel 0.22 #​4445 (rtyler)
• fix: allow partition column changes on schema overwrite #​4444 (ethan-tyler)
• refactor: remove DeltaTableProvider #​4435 (ethan-tyler)
• test: port DeltaTableProvider tests to next provider #​4432 (ethan-tyler)
• refactor: clean up row index scan contract #​4431 (ethan-tyler)
• chore: clarify cached stats arrow fields #​4429 (ethan-tyler)
• fix: reduce duplicate validation memory #​4428 (ethan-tyler)
• fix(core): reject unsupported column mapping writes #​4424 (fksegundo)
• refactor: selective stats materialization policy #​4421 (ethan-tyler)
• fix: local file table root identity #​4419 (ethan-tyler)
• ci: build wheels with 64k pages #​4418 (rtyler)
• chore: upgrade to the 200 series of buoyant_kernel #​4416 (rtyler)
• chore: update the changelog and prepare for the next patch release #​4415 (rtyler)
• fix: exclude partition columns from add stats schema #​4408 (ethan-tyler)
• feat: nanosecond timestamps support, gated by a Cargo feature #​4370 (itamarst)
• fix: propagate schema_force_view_types through merge write projection #​4302 (sheganinans)

v0.32.1

Full Changelog

Closed issues:

• [Bug]: Regression in 1.5.1: schema_mode='overwrite' with predicate fails for mixed-case column names (utcDate) #​4404
• [Bug]: update_incremental() and transaction_version() hold the GIL for the entire S3 round-trip #​4402
• [Feature]: Build Musl arm64 wheels #​4392
• [Bug]: deltalake_catalog_unity failes to list catalogs #​4360
• [Bug]: UnityCatalogFactory::parse_url_opts only allows unity-specific options #​4225

Merged pull requests:

• chore: cap the upper range of the buoyant_kernel for this release #​4412 (rtyler)
• fix(catalog-unity): skip non-Unity keys in try_with_options #​4410 (1fanwang)
• fix: listing catalogs from Unity #​4409 (hjohnss6)
• fix: preserve mixed case columns in replaceWhere writes #​4407 (ethan-tyler)
• fix: release GIL for DeltaTable update calls #​4406 (ethan-tyler)
• chore: add tests for cleanup_expired_logs_for #​4405 (mbkroese)
• refactor: introduce selective stats projection primitives #​4403 (ethan-tyler)
• chore: regenerate updated changelog #​4400 (rtyler)
• feat: add aarch64 musl wheel builds #​4399 (Slyrc)
• chore: next iteration of development #​4396 (rtyler)
• fix: allow boolean to exist in our stats schema #​4395 (rtyler)
• chore: bump python for next release #​4393 (rtyler)

v0.32.0

Full Changelog

Implemented enhancements:

• feature: Add post_commithook_properties to DeltaTable.restore #​4251
• Create deltalake-engine crate based off arrow engine #​4133

Fixed bugs:

• [Bug]: DeltaScan ignores DeltaScanConfig::schema unlike old DeltaTableProvider #​4306
• bug: Streamed merge loses file pruning for string partition IN (...) predicates #​4301

Closed issues:

• [Bug]: append fails when generated column uses TRUNC #​4386
• [Bug]: num_deleted_rows is 0 for partition-only DELETEs #​4379
• [Bug]: DeltaTable.to_pyarrow_dataset() hangs indefinitely with moto S3 mock (regression in 1.3.2) #​4362
• [Bug]: Number of deleted rows incorrect when table had "enableChageDataFeed" set to true #​4353
• [Bug]: MERGE on partitioned table produces duplicate rows when partition value contains a space #​4352
• Datafusion/Arrow/Delta Kernel Upgrade #​4344
• [Bug]: intermittent S3 integration test failures in CI #​4332
• [Bug]: az:// URL scheme causes path inconsistency between log store and kernel, leading to 'version already exists' errors on Azure #​4320
• Documentation for Remove::data_change does not match spec #​4317
• [Bug]: Wrong log store registry for file:// schema #​4316
• [Bug]: Insertion Order not maintained when compacting #​4310
• [Bug]: DeltaScan scan includes file column although it's not projected #​4307
• [Bug]: Unable to write on GCP bucket in S3 compatibility mode #​4303
• [Bug]: merge_schema=True fails on first merge into fresh table partitioned by string column #​4298
• Deduplicate _last_checkpoint parsing in deltalake-core #​4294
• Inconsistent i64/u64 usage for defining versions #​4293
• [Bug]: 1.5.0 regression a partitioned Delta table with existing data and a merge into it with merge_schema: True #​4292
• [Bug]: Vacuuming FULL after compaction potentially removes data files preventing time travel #​4290
• [Bug]: Vacuuming with keep_versions requires versions to be in sorted order #​4286
• bug: same version snapshot update doesn't adopt checkpoint written later #​4280
• Re-use stored state in EagerSnapshot #​4269
• [Bug]: Datafusion not reading all partitions on Linux machine #​4268
• [Feature]: CommitData::new should respect "clientVersion" from app_metadata #​4263
• delta_kernel 0.20 upgrade #​4256
• [Feature]: add some changelog content to the docs #​4253
• feature: make commit control args keyword only across mutating APIs #​4252
• Passing cert file in python storage_options #​4246
• [Bug]: After lazy writes, writing large files fails #​4229
• [Bug]: deltalake get_add_actions not exposing bool min/max #​4224
• [Bug]: slack link in issue #​4214

v0.31.1

Full Changelog

Implemented enhancements:

• Use batched delete in DeltaStorageHandler.delete_dir #​4243
• [Feature]: refactor DeltaDataSink to go through delta writer builder #​4189

Fixed bugs:

• [Bug]: Read delta table from s3 directory bucket #​4136

Closed issues:

• [Bug]: update_incremental is not incremental #​4276
• [Bug]: pub fn create_add signature includes types that are not pub use #​4273
• [Feature]: Support for Writing to Delta Tables with Identity Columns in delta-rs #​4237
• [Bug]: Python delection_vectors() truncates bool list. #​4235
• [Feature]: Vacuum Lite to avoid unnecessary file listing #​4228
• [Feature]: Write incrementally to table when merging #​4217
• [Feature]: change existing table's enableChangeDataFeed configuration to be updated to True #​4216
• [Bug]: Contributing guide could be improved #​4213
• [Docs]: usage/appending-overwriting-delta-lake-table has broken language tabs #​4200
• [Bug]: s3-native-tls feature still brings aws-lc-rs dependency #​4170
• bug: generated column expr fails when SchemaMode::Merge would add referenced column #​4169
• [Bug]: create_write_transaction no longer overwrites partitions #​4126

v0.31.0

Full Changelog

Implemented enhancements:

• update_datafusion_session has the wrong order of arguments #​4139

Closed issues:

• [Bug]: write_deltalake with schema_mode="merge" drops generated column metadata #​4186
• [Bug]: is_deltatable creates paths for non existing delta table location #​4175
• [Bug]: Rust CdfLoadBuilder.build function is crate private and prohibits using feature #​4160
• Expose deletion vectors information to Python #​4159
• [Bug]: polars.DataFrame.write_delta() failing for unknown argument 'schema' #​4157
• [Bug]: Delete regression in 1.4.0: partition‑only predicate removes too few files #​4149
• [Bug]: Unity Catalog Azure credentials error #​4129

Merged pull requests:

• chore: enable snappy compression on checkpoints #​4209 (rtyler)
• chore: set compression for partition optimization as well #​4208 (rtyler)
• chore: change the versions for the next "majorish" release of 🦀 #​4207 (rtyler)
• fix: enforce file-id filter semantics in scan planning #​4206 (ethan-tyler)
• refactor(python): get add action return arrow table #​4204 (vsmanish1772)
• fix: propagate session config through Delta factory path #​4202 (ethan-tyler)
• chore: drop unused ReceiverStreamBuilder spawn #​4201 (ethan-tyler)
• chore: harden scan adapter caching and DV mask edge cases #​4199 (ethan-tyler)
• fix(datafusion): avoid overflow when scanning add actions #​4197 (vsmanish1772)
• fix: clarify vacuum command documentation for DeltaTable #​4196 (khalidmammadov)
• refactor: use BatchAdapterFactory for scan adaptation #​4195 (ethan-tyler)
• perf: parallel partition writers via per-stream JoinSet #​4193 (fvaleye)
• fix: make session the first argument of update_datafusion_session #​4192 (pauldouane)
• fix: preserve generated column metadata during schema merge #​4191 (ethan-tyler)
• refactor: use FileSelection for matched file scans #​4188 (ethan-tyler)
• feat: add DeltaScan insert_into with runtime log_store #​4187 (ethan-tyler)
• chore: bump version from 1.4.1 to 1.4.2 #​4182 (ion-elgreco)
• fix: is_deltatable must not to create paths for not existing tables paths #​4176 (khalidmammadov)
• refactor: remove table level stats on TableProvider #​4174 (roeap)
• docs: add storage backend configuration reference tables #​4173 (immohamedadhil)
• refactor: add FileSelection to next provider for DeltaTableProvider removal #​4172 (ethan-tyler)
• feat: expose DV metadata and payloads as Arrow streams #​4168 (ethan-tyler)
• fix: align file_id with DataFusion UInt16 dictionary #​4167 (ethan-tyler)
• chore: add the unity catalog dependency back #​4165 (hntd187)
• fix: preserve kernel column segments #​4164 (ethan-tyler)
• chore: use default-https-client for aws sdk to avoid deps on hyper 0.14 #​4163 (BugenZhao)
• fix: unblock schema merge appends with generated columns #​4162 (ethan-tyler)
• fix(cdf): make cdf builders build function accessible #​4161 (khalidmammadov)
• fix(docs): open_table examples to use URLs instead of str #​4154 (khalidmammadov)
• perf: cache schema per stream instead of per batch #​4152 (fvaleye)
• refactor: avoid mutable updates of inner snapshot #​4151 (roeap)
• fix(delete): use Add metadata for partition only DELETE #​4150 (ethan-tyler)
• fix: nested runtimes in stream adapter #​4148 (ion-elgreco)
• feat: session-first DataFusion integration + session resolution policies #​4145 (ethan-tyler)
• fix: align stats to view typed schema + harden parquet predicate pushdown #​4144 (ethan-tyler)
• fix(python): guard DataFusion FFI export on datafusion major version #​4142 (ethan-tyler)
• chore: upgrade python version for a patch release #​4141 (rtyler)
• feat: update asserted nullability in DataValidation output schema #​4132 (roeap)
• docs: incorporate some AI guidance for contributors #​4131 (rtyler)
• fix(core): align file stats with parquet read schema #​4130 (roeap)
• fix(datafusion): resolve DML predicates against execution scan schema #​4127 (ethan-tyler)
• chore: upgrade python version for the next release #​4124 (rtyler)
• chore: allow integration tests to be run in parallel with nextest #​4122 (rtyler)
• chore: upgrade azurite and purge the need for a local az CLI to run tests #​4121 (rtyler)
• feat: migrate delete #​4117 (roeap)

v0.30.2

Full Changelog

Fixed bugs:

• [Bug]: DeltaTable.delete fails during equality comparison with a string #​4125

Closed issues:

• [Bug]: write_deltalake errors on missing nullable columns without generation expressions #​4156
• Bug Report: Nested Tokio Runtime Panic in write_deltalake #​4147
• [Bug]: Segmentation fault when registering table to datafusion session #​4135
• [Bug]: slack link no longer active #​4118

v0.30.1

Full Changelog

v0.30.0

⚠️ There are a number of API changes between 0.30.x and 0.29.4, but we expect to see better performance as a result!

Full Changelog

Merged pull requests:

• refactor: remove log_data call sites in find_files #​4026 (roeap)
• chore: remove wildcard dependency for publishing #​4025 (rtyler)
• refactor: use logical type ref when getting stats #​4019 (roeap)
• fix: handle stats config in data sink #​4016 (roeap)
• fix: null handling when extracting scalars #​4014 (roeap)
• fix: between range handling in expression translations #​4013 (roeap)
• chore: fix windows uri test #​4011 (hntd187)
• refactor: towards lazier snapshots #​4010 (roeap)
• fix: pin pyspark and clear disk space in runners #​4007 (ion-elgreco)
• test: add utilities for asserting DAT scan results #​4005 (roeap)
• chore: update delta-kernel to 0.19 #​4004 (roeap)
• refactor: simplify kernel extensions #​4003 (roeap)
• chore: clippy #​4002 (roeap)
• refactor: handle target version when resolving snapshot #​4001 (roeap)
• refactor: use rstest for running DAT tests #​4000 (roeap)
• feat: kernel expression conversion #​3998 (roeap)
• chore: add easier local coverage reporting #​3995 (rtyler)
• feat: expose operations on DeltaTable #​3987 (roeap)
• chore: remove some warnigs #​3986 (roeap)
• chore: normalize Url going into logstore and update everything to take references #​3985 (rtyler)
• fix: add missing field to snapshot serde #​3984 (roeap)
• feat: allow for concurrent deletes in conflict checker if data_change is false #​3982 (abhiaagarwal)
• fix: remove 3.9 from ci matrix #​3978 (ion-elgreco)
• fix: decode path before lookup #​3976 (ion-elgreco)
• chore: remove deprecated pyo3 methods #​3975 (ion-elgreco)
• chore: removing APIs and deprecation warnings: 0.30.x here we come #​3962 (rtyler)
• feat: update to DataFusion 51, arrow 57, delta-kernel 0.18.0, pyo3 26, pyo3-arrow 0.14 #​3949 (hntd187)
• fix: schema evolution for merge operation #​3945 (JustinRush80)
• chore: remove Python 3.9 from our build infrastructure #​3937 (rtyler)
• docs: fix small typo issue #​3935 (bmoreau8)
• chore: removing references to using partition_filters for partition overwrite #​3912 (zyd14)
• feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #​3847 (fvaleye)

Fixed bugs:

• Asked to increase max_temp_directory_size in the disk manager configuration when optimizing large table #​3833

Closed issues:

• [Bug]: Count / get_add_actions exception for an empty table #​4023
• [Bug]: MERGE with schema evolution does not add new columns #​4009
• [Bug]: vacuum does not respect retention_hours when full=True #​3989
• [Bug]: write table by FFI call from go may memory leak? #​3973
• [Bug]: Table merging fails with merge_schema=True #​3943
• [Bug]: _internal.DeltaError: Generic DeltaTable error: Unable to map __delta_rs_path to action during overwrite with predicate #​3939
• [Feature]: update to DataFusion 51.0.0 #​3920
• [Bug]: get_add_actions() panics with "index out of bounds" when table has no data files #​3918
• [Bug]: Docs describe partition_filters parameter to write_deltalake that doesn't exist #​3904
• [Feature]: split delta-rs into multiple crates #​3899
• [Feature]: Drop python 3.9 support once EOL #​3886
• [Bug]: PyPi storage limit hit for deltalake [python releases blocked for time-being] #​3876

apache/arrow-rs-object-store (object_store)

v0.13.2

Compare Source

Full Changelog

Implemented enhancements:

• Path::join(Self, &PathPart) -> Self #​665
• Support for AWS Encryption Client encryption #​647
• LocalFileSystem: use read_at instead of seek + read #​622
• Avoid reading metadata for LocalFileSystem::read_ranges (and other methods) #​614
• expose Inner from HttpRequestBody #​606
• Release object store 0.13.1 (maintenance) - Target Jan 2026 #​598
• Support AWS_ENDPOINT_URL_S3 in aws backend #​589
• Release object store 0.12.5 (maintenance) - Target Dec 2025 #​582
• Support upper-case configuration options in parse_url_opts #​529
• object_store: Support {az,abfs,abfss}://container@account.blob.{core.windows.net,fabric.microsoft.com} URLs #​430
• Support Transfer-Encoding: chunked responses in HttpStore #​340
• Use reconstructed ListBlobs marker to provide list offset support in MicrosoftAzure store #​461

Fixed bugs:

• Azure Fabric: Unsigned integer underflow when fetching token causes integer overflow panic #​640
• Error body missing for 5xx errors after retry exhausted #​617
• Heavy contention on credentials cache #​541
• AWS/S3 Default Headers are not considered for signature calculation #​484
• az:// <container> not work as expected #​443

Performance improvements:

• Preallocate single Vec in get_ranges for LocalFilesystem #​634
• Use platform specific read_at when available #​628 (AdamGS)
• Avoid metadata lookup for LocalFileSystem::read_ranges and chunked_stream #​621 (Dandandan)

Closed issues:

• [Security Alert] Exposed API key(s) detected: AWS Access Key #​659
• AWS S3 token expired on multi-threaded app with Arc usage #​655
• Emulator tests fail in CI due to an unsupported service version header in Azurite #​626

Merged pull requests:

• Replace Path::child with Path::join #​666 (Kinrany)
• Support --xa-s3 suffix for S3 Express One Zone bucket access points #​663 (pdeva)
• docs: clarify Clone behavior #​656 (crepererum)
• Implement Clone for local and memory stores #​653 (DoumanAsh)
• Unify from_env behaviours #​652 (miraclx)
• docs: add examples to the aws docs where appropriate #​651 (CommanderStorm)
• Switch TokenCache to RWLock #​648 (tustvold)
• Minimize futures dependency into relevant sub-crates #​646 (AdamGS)
• Clarify ShuffleResolver doc-comments #​645 (jkosh44)
• Introduce a "tokio" to allow pulling a trait-only build #​644 (AdamGS)
• fix(azure): fix integer overflow in Fabric token expiry check #​641 (desmondcheongzx)
• chore: upgrade to rand 0.10 #​637 (crepererum)
• fix(aws): Include default headers in signature calculation (#​484) #​636 (singhsaabir)
• fix(azure): correct Microsoft Fabric blob endpoint domain #​631 (kevinjqliu)
• Unblock emulator based tests #​627 (AdamGS)
• Azure ADLS list_with_offset support #​623 (omar)
• Implement tests for range and partial content responses #​619 (vitoordaz)
• fix: missing 5xx error body when retry exhausted #​618 (jackye1995)
• build(deps): update nix requirement from 0.30.0 to 0.31.1 #​616 (dependabot[bot])
• Clarify behavior of parse_url_opts with regards to case sensitivity #​613 (AdamGS)
• Fix logical format conflict #​605 (tustvold)
• Fix Azure URL parsing #​604 (tustvold)
• build(deps): update quick-xml requirement from 0.38.0 to 0.39.0 #​602 (dependabot[bot])
• Only read file metadata once in LocalFileSystem::read_ranges #​595 (AdamGS)
• feat: Add support for AWS_ENDPOINT_URL_S3 environment variable #​590 (rajatgoel)
• feat: impl MultipartStore for PrefixStore #​587 (ddupg)
• Implement typos-cli #​570 (jayvdb)
• feat (azure): support for '.blob.core.windows.net' in "az://" scheme #​431 (vladidobro)

* This Changelog was automatically generated by github_changelog_generator

v0.13.1

Compare Source

v0.13.0

Compare Source

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.