Skip to content

Commit 0841f54

Browse files
author
“Zhibin
committed
docs: Add missing parameter details in add_files documentatio (#2132)
1 parent 045dd10 commit 0841f54

1 file changed

Lines changed: 48 additions & 8 deletions

File tree

mkdocs/docs/api.md

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1004,6 +1004,30 @@ To show only data files or delete files in the current snapshot, use `table.insp
10041004

10051005
Expert Iceberg users may choose to commit existing parquet files to the Iceberg table as data files, without rewriting them.
10061006

1007+
<!-- prettier-ignore-start -->
1008+
1009+
!!! note "Name Mapping"
1010+
Because `add_files` uses existing files without writing new parquet files that are aware of the Iceberg's schema, it requires the Iceberg's table to have a [Name Mapping](https://iceberg.apache.org/spec/?h=name+mapping#name-mapping-serialization) (The Name mapping maps the field names within the parquet files to the Iceberg field IDs). Hence, `add_files` requires that there are no field IDs in the parquet file's metadata, and creates a new Name Mapping based on the table's current schema if the table doesn't already have one.
1011+
1012+
!!! note "Partitions"
1013+
`add_files` only requires the client to read the existing parquet files' metadata footer to infer the partition value of each file. This implementation also supports adding files to Iceberg tables with partition transforms like `MonthTransform`, and `TruncateTransform` which preserve the order of the values after the transformation (Any Transform that has the `preserves_order` property set to True is supported). Please note that if the column statistics of the `PartitionField`'s source column are not present in the parquet metadata, the partition value is inferred as `None`.
1014+
1015+
!!! warning "Maintenance Operations"
1016+
Because `add_files` commits the existing parquet files to the Iceberg Table as any other data file, destructive maintenance operations like expiring snapshots will remove them.
1017+
1018+
<!-- prettier-ignore-end -->
1019+
1020+
### Usage
1021+
1022+
| Parameter | Required? | Type | Description |
1023+
| ------------------------- | --------- | -------------- | ----------------------------------------------------------------------- |
1024+
| `file_paths` | ✔️ | List[str] | The list of full file paths to be added as data files to the table |
1025+
| `snapshot_properties` | | Dict[str, str] | Properties to set for the new snapshot. Defaults to an empty dictionary |
1026+
| `check_duplicate_files` | | bool | Whether to check for duplicate files. Defaults to `True` |
1027+
1028+
### Example
1029+
1030+
Add files to Iceberg table:
10071031
```python
10081032
# Given that these parquet files have schema consistent with the Iceberg table
10091033
@@ -1019,18 +1043,34 @@ tbl.add_files(file_paths=file_paths)
10191043
# A new snapshot is committed to the table with manifests pointing to the existing parquet files
10201044
```
10211045

1022-
<!-- prettier-ignore-start -->
1046+
Add files to Iceberg table with custom snapshot properties:
1047+
```python
1048+
# Assume an existing Iceberg table object `tbl`
10231049

1024-
!!! note "Name Mapping"
1025-
Because `add_files` uses existing files without writing new parquet files that are aware of the Iceberg's schema, it requires the Iceberg's table to have a [Name Mapping](https://iceberg.apache.org/spec/?h=name+mapping#name-mapping-serialization) (The Name mapping maps the field names within the parquet files to the Iceberg field IDs). Hence, `add_files` requires that there are no field IDs in the parquet file's metadata, and creates a new Name Mapping based on the table's current schema if the table doesn't already have one.
1050+
file_paths = [
1051+
"s3a://warehouse/default/existing-1.parquet",
1052+
"s3a://warehouse/default/existing-2.parquet",
1053+
]
10261054

1027-
!!! note "Partitions"
1028-
`add_files` only requires the client to read the existing parquet files' metadata footer to infer the partition value of each file. This implementation also supports adding files to Iceberg tables with partition transforms like `MonthTransform`, and `TruncateTransform` which preserve the order of the values after the transformation (Any Transform that has the `preserves_order` property set to True is supported). Please note that if the column statistics of the `PartitionField`'s source column are not present in the parquet metadata, the partition value is inferred as `None`.
1055+
# Custom snapshot properties
1056+
snapshot_properties = {"abc": "def"}
10291057

1030-
!!! warning "Maintenance Operations"
1031-
Because `add_files` commits the existing parquet files to the Iceberg Table as any other data file, destructive maintenance operations like expiring snapshots will remove them.
1058+
# Enable duplicate file checking
1059+
check_duplicate_files = True
10321060

1033-
<!-- prettier-ignore-end -->
1061+
# Add the Parquet files to the Iceberg table without rewriting
1062+
tbl.add_files(
1063+
file_paths=file_paths,
1064+
snapshot_properties=snapshot_properties,
1065+
check_duplicate_files=check_duplicate_files
1066+
)
1067+
1068+
# NameMapping must have been set to enable reads
1069+
assert tbl.name_mapping() is not None
1070+
1071+
# Verify that the snapshot property was set correctly
1072+
assert tbl.metadata.snapshots[-1].summary["abc"] == "def"
1073+
```
10341074

10351075
## Schema evolution
10361076

0 commit comments

Comments
 (0)