+The `check_duplicate_files` parameter is `True` by default and will check the new files against the existing Iceberg table data files to prevent duplicates. This check can be expensive for large tables with many files. It is recommended to use the default configuration. The check can be turned off by setting `check_duplicate_files=False`, but this may result in duplicate files being added to the table, which can lead to data consistency issues and potential table corruption if the same data file is added multiple times.
0 commit comments