Connector
Athena
Feature area
Profiler / Data Quality
Describe the bug
When using customTestDefinition with the Jinja template variable {{ table_name }} against an AWS Athena service, OpenMetadata evaluates {{ table_name }} to the full OpenMetadata FQN (e.g., default.my_database.my_table)
AWS Athena does not recognize default. as a valid catalog prefix, causing queries to fail with syntax parsing errors. The engine should strip the service entity name or translate it to database.table / AwsDataCatalog.database.table.
To Reproduce
Create a test definition with a Custom SQL query, such as
SELECT COUNT(*) FROM {{ table_name }} WHERE is_nan({{ column_name }} )
Use it on a Athena table column (numeric in my case, but anything else will make the job), and run the test pipeline.
You will get an Athena error due to the prefix (which is default by default).
Expected behavior
{{ table_name }} for an Athena connector should resolve strictly to schema.table (e.g., my_schema.my_table) or use the appropriate catalog prefix AwsDataCatalog.my_schema.my_table.
Connection / ingestion config
Logs
OS
No response
Python version
No response
OpenMetadata version
2.0.0
OpenMetadata Ingestion package version
No response
Additional context
I found a workaround by using Jinja filtering in the custom SQL. In my case :
SELECT COUNT(*) FROM {{ table_name.split('.')[-2:] | join('.') }} WHERE is_nan({{ column_name }} )
will work as this will drop the prefix. But I do not think this is the expected behaviour. I opened an enhancement request (#30749 ) a couple days ago, but after further consideration I consider this as a bug.
Pre-submission checklist
Connector
Athena
Feature area
Profiler / Data Quality
Describe the bug
When using customTestDefinition with the Jinja template variable {{ table_name }} against an AWS Athena service, OpenMetadata evaluates {{ table_name }} to the full OpenMetadata FQN (e.g., default.my_database.my_table)
AWS Athena does not recognize default. as a valid catalog prefix, causing queries to fail with syntax parsing errors. The engine should strip the service entity name or translate it to database.table / AwsDataCatalog.database.table.
To Reproduce
Create a test definition with a Custom SQL query, such as
SELECT COUNT(*) FROM {{ table_name }} WHERE is_nan({{ column_name }} )Use it on a Athena table column (numeric in my case, but anything else will make the job), and run the test pipeline.
You will get an Athena error due to the prefix (which is default by default).
Expected behavior
{{ table_name }}for an Athena connector should resolve strictly to schema.table (e.g., my_schema.my_table) or use the appropriate catalog prefix AwsDataCatalog.my_schema.my_table.Connection / ingestion config
Logs
OS
No response
Python version
No response
OpenMetadata version
2.0.0
OpenMetadata Ingestion package version
No response
Additional context
I found a workaround by using Jinja filtering in the custom SQL. In my case :
SELECT COUNT(*) FROM {{ table_name.split('.')[-2:] | join('.') }} WHERE is_nan({{ column_name }} )will work as this will drop the prefix. But I do not think this is the expected behaviour. I opened an enhancement request (#30749 ) a couple days ago, but after further consideration I consider this as a bug.
Pre-submission checklist