Invalid Input Error: Function with name "read_avro" not found in ExtensionUtil::GetTableFunction

[Fokko]

Since yesterday we've noticed an issue with the PyIceberg CI when we do integration testing with PyIceberg:

    @pytest.mark.integration
    def test_duckdb_url_import(warehouse: Path, arrow_table_with_null: pa.Table) -> None:
        os.environ["TZ"] = "Etc/UTC"
        time.tzset()
        tz = pytz.timezone(os.environ["TZ"])
    
        catalog = SqlCatalog("test_sql_catalog", uri="sqlite:///:memory:", warehouse=f"/{warehouse}")
        catalog.create_namespace("default")
    
        identifier = "default.arrow_table_v1_with_null"
        tbl = _create_table(catalog, identifier, {}, [arrow_table_with_null])
        location = tbl.metadata_location
    
        import duckdb
    
        duckdb.sql("INSTALL iceberg; LOAD iceberg;")
>       result = duckdb.sql(
            f"""
        SELECT *
        FROM iceberg_scan('{location}')
        """
        ).fetchall()
E       duckdb.duckdb.InvalidInputException: Invalid Input Error: Function with name "read_avro" not found in ExtensionUtil::GetTableFunction

Does this sounds a bell?

[kevinjqliu]

Its using the read_avro function from the avro extension
https://github.com/search?q=repo%3Aduckdb%2Fduckdb-iceberg+ExtensionUtil%3A%3AGetTableFunction+read_avro&type=code

I'd assume its auto loaded. This was recently changed by #110

[kevinjqliu]

Looks like we need to add auto load avro extension here

duckdb-iceberg/src/iceberg_extension.cpp

Lines 238 to 242 in 94d95da

	ExtensionHelper::AutoLoadExtension(instance, "parquet");
	if (!instance.ExtensionIsLoaded("parquet")) {
	throw MissingExtensionException("The iceberg extension requires the parquet extension to be loaded!");
	}

[kevinjqliu]

or do hot auto loading like this

duckdb-iceberg/src/iceberg_extension.cpp

Lines 130 to 134 in 94d95da

	if (!__AVRO_LOADED__) {
	auto &db_instance = context.db;
	ExtensionHelper::AutoLoadExtension(*db_instance, "avro");
	__AVRO_LOADED__ = true;
	}

[Tishj]

Yea this sounds like what we have in our tests:

require avro

require parquet

require iceberg

It essentially loads the extensions or fails (skips the test) if the extension isn't built/available

[kevinjqliu]

I think this should be fixed internally by the duckdb-iceberg extension. Otherwise, clients have to always do INSTALL avro; LOAD avro; when using this extension

require avro

Does require here mean avro is installed or loaded or both?

[Tmonster]

Avro is auto loaded when a user creates an ICEBERG secret or attaches an ICEBERG catalog so they don't have to call INSTALL avro; LOAD avro;

I'm pretty sure the require avro is still needed to avoid introducing any side effects that may be introduced by auto loading extensions

[Tmonster]

I've managed to reproduce this error, will work on fixing the extension. Sorry for the headaches/broken tests

[Fokko]

@Tmonster No problem at all, appreciate the quick replies 🙌

[Tmonster]

We just published an updated version of the extension. The errors should go away if the new iceberg extension is downloaded (hash c6efd60). Feel free to reopen if this is not the case

[kevinjqliu]

Looks like the issue is fixed on the pyiceberg side apache/iceberg-python#1927
Thanks @Tmonster