Rewrite manifests #1661

Closed

amitgilad3 wants to merge 112 commits intoapache:mainfrom

amitgilad3:rewrite_manifests

Commits on May 17, 2025

Commits on Jul 7, 2025

Replace Deprecated (Current) OAuth2 Handling with AuthManager Implementation `LegacyOAuth2AuthManager` (#1981)

sungwy
authored and
amitgilad3
committed
Add Hugging Face filesystem support to fsspec (#1997)

lhoestq
authored and
amitgilad3
committed
Build: Bump flask-cors from 5.0.1 to 6.0.0 (#2012)

dependabot[bot]
authored and
amitgilad3
committed
Add ADLFS property `adls.account-host` (#2016)

vsdsantos
authored and
amitgilad3
committed
Build: Bump setuptools from 78.1.0 to 78.1.1 (#2027)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump fastavro from 1.10.0 to 1.11.1 (#2026)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump sqlalchemy from 2.0.40 to 2.0.41 (#2025)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump getdaft from 0.4.14 to 0.4.15 (#2023)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.12 to 1.38.18 (#2021)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.31.2 to 0.31.4 (#2020)

dependabot[bot]
authored and
amitgilad3
committed
feat: `validate_deleted_data_files` (#1938)

authored and
amitgilad3
committed
Build: Bump mkdocs-material from 9.6.13 to 9.6.14 (#2024)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump cython from 3.0.12 to 3.1.1 (#2022)

dependabot[bot]
authored and
amitgilad3
committed
Fix the CI (#2049)

Fokko
authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.18 to 1.38.22 (#2038)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump moto from 5.1.4 to 5.1.5 (#2039)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pytest-mock from 3.14.0 to 3.14.1 (#2040)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump getdaft from 0.4.15 to 0.4.16 (#2042)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pydantic from 2.11.4 to 2.11.5 (#2045)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump coverage from 7.8.0 to 7.8.2 (#2047)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.31.4 to 0.32.1 (#2046)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump thrift from 0.21.0 to 0.22.0 (#2043)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mkdocstrings-python from 1.16.10 to 1.16.11 (#2044)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mkdocs-autorefs from 1.4.1 to 1.4.2 (#2048)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump cachetools from 5.5.2 to 6.0.0 (#2041)

dependabot[bot]
authored and
amitgilad3
committed
Add `.vscode` folder to `.gitignore` (#2053)

rambleraptor
authored and
amitgilad3
committed
Make `validation_history` private before release (#2054)

sungwy
authored and
amitgilad3
committed
Build: Bump datafusion from 46.0.0 to 47.0.0 (#2063)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.32.2 to 0.32.3 (#2061)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump getdaft from 0.4.16 to 0.4.18 (#2060)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pyspark from 3.5.5 to 3.5.6 (#2062)

dependabot[bot]
authored and
amitgilad3
committed
Leverage Iceberg-Rust for all the transforms (#1833)

authored and
amitgilad3
committed
Build: Bump cython from 3.1.1 to 3.1.2 (#2081)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.32.3 to 0.32.4 (#2077)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump requests from 2.32.3 to 2.32.4 (#2078)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump duckdb from 1.2.2 to 1.3.0 (#2080)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pandas from 2.2.3 to 2.3.0 (#2079)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mkdocstrings-python from 1.16.11 to 1.16.12 (#2076)

dependabot[bot]
authored and
amitgilad3
committed
Robustify boto3 session handling (DynamoDB, RestCatalog) (#2071)

authored and
amitgilad3
committed
chore: remove non-test asserts (#2082)

jayceslesar
authored and
amitgilad3
committed
fix: glue drop_namespace to check non-iceberg tables (#2083)

geruh
authored and
amitgilad3
committed
Add v3 structs (#2065)

b-phi
authored and
amitgilad3
committed
Add Avro compression (#1976)

Fokko
authored and
amitgilad3
committed
REST: Pass `purgeRequest` to catalog (#2086)

jayceslesar
authored and
amitgilad3
committed
Bump Ruff to the latest version (#2073)

Fokko
authored and
amitgilad3
committed
Bump mypy in pre-commit (#2074)

Fokko
authored and
amitgilad3
committed
feat: support listing known catalogs (#2088)

Anton-Tarazi
authored and
amitgilad3
committed
fix: add metadata_properties to _construct_parameters when update hive table (#2013)

kadai0308
authored and
amitgilad3
committed
change catalog default warehouse location to not use hive-style warehouse location (#2059)

kevinjqliu
authored and
amitgilad3
committed
Bump PyIceberg to 0.9.1 (#2092)

Fokko
authored and
amitgilad3
committed
Bump tests to Spark 3.5.6 (#2091)

Fokko
authored and
amitgilad3
committed
Bump Iceberg-Java to 1.9.1 (#2093)

authored and
amitgilad3
committed
maint: use `URI` constant instead of 'uri' strings (#2094)

jayceslesar
authored and
amitgilad3
committed
Build: Bump pypa/cibuildwheel from 2.23.3 to 3.0.0 (#2099)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump moto from 5.1.5 to 5.1.6 (#2100)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump cachetools from 5.5.2 to 6.1.0 (#2101)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.32.5 to 0.33.0 (#2103)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump typing-extensions from 4.13.2 to 4.14.0 (#2104)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump duckdb from 1.3.0 to 1.3.1 (#2105)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump boto3 from 1.37.3 to 1.38.27 (#2108)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pydantic from 2.11.5 to 2.11.7 (#2107)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump coverage from 7.8.2 to 7.9.1 (#2106)

dependabot[bot]
authored and
amitgilad3
committed
Docs: fix manifest min count to merge description (#2116)

ThomasDsantos
authored and
amitgilad3
committed
Prefer `FileIO` over the PyArrow `FileSystem` (#2115)

Fokko
authored and
amitgilad3
committed
Add `ExpireSnapshots` (#1880)

authored and
amitgilad3
committed
Build: Bump third party deps (#2127)

Fokko
authored and
amitgilad3
committed
Support ADLS with Pyarrow file IO (#2111)

authored and
amitgilad3
committed
Improve upsert memory pressure (#1995)

authored and
amitgilad3
committed
Type mismatch Avro zstd decompression (#2128)

zhongyujiang
authored and
amitgilad3
committed
Expose S3 retry strategy (#2110)

Fokko
authored and
amitgilad3
committed
Refactor `Makefile` and run s3/adls/gcs tests (#2125)

kevinjqliu
authored and
amitgilad3
committed
perf: optimize `table.add_files` and `inspect.files` (#2133)

authored and
amitgilad3
committed
Run `test_create_table_transaction` for Rest V1 tables (#2137)

Fokko
authored and
amitgilad3
committed
Add warehouse parameter to the REST Catalog doc (#2066)

authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.22 to 1.38.42 (#2144)

dependabot[bot]
authored and
amitgilad3
committed
test: use `create or replace` (#2140)

kevinjqliu
authored and
amitgilad3
committed
Update assertion for testing locally (#2113)

kevinjqliu
authored and
amitgilad3
committed
Validate added data files for snapshot compatibility (#2050)

authored and
amitgilad3
committed
Add `hive.kerberos-service-name` configuration option (#2032) (#2141)

authored and
amitgilad3
committed
Relax REST client `/v1/config` response constraints (#2148)

authored and
amitgilad3
committed
dep: update `multidict`, previous version yanked (#2157)

kevinjqliu
authored and
amitgilad3
committed
Build: Bump mypy-boto3-dynamodb from 1.38.4 to 1.39.0 (#2163)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump mypy-boto3-glue from 1.38.42 to 1.39.0 (#2164)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump huggingface-hub from 0.33.0 to 0.33.1 (#2165)

dependabot[bot]
authored and
amitgilad3
committed
Build: Bump pyroaring from 1.0.1 to 1.0.2 (#2162)

dependabot[bot]
authored and
amitgilad3
committed
Feature: Write to branches (#941)

authored and
amitgilad3
committed
Hive: update hive storage descriptor after commit schema change (#2036)

frankliee
authored and
amitgilad3
committed
fix: Propagate Table properties to Glue (#2097)

summermousa-vendia
authored and
amitgilad3
committed
Documented `row_filter` expressions (#1862)

authored and
amitgilad3
committed
Fix the CI (#2171)

Fokko
authored and
amitgilad3
committed
add iceberg datafusion integration (#2075)

kevinjqliu
authored and
amitgilad3
committed
Cleanup in `expression-dsl.md` (#2168)

Fokko
authored and
amitgilad3
committed
Docs: Condens Python API docs (#2139)

authored and
amitgilad3
committed
feat: add support for setting and removing table properties on console (#2153)

blinkseb
authored and
amitgilad3
committed
first attempt at init_manifests
amitgilad3
committed
initial implementation of rewrite_manifests
amitgilad3
committed
fix lint
amitgilad3
committed
1. support returning rewrite results
amitgilad3
committed
lint and rebase
amitgilad3
committed
linting errors
amitgilad3
committed

Commits

Commits on May 15, 2025

Commits on May 17, 2025

Commits on Jul 7, 2025