You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7e299cc
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/integrations/engines/clickhouse.md
+52Lines changed: 52 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -470,6 +470,57 @@ gateways:
470
470
471
471
With this configuration, ClickHouse models will appear as `ch_virtual.mydb.mytable` in plan output instead of `__clickhouse__.mydb.mytable`.
472
472
473
+
### Reading catalog-backed physical tables
474
+
475
+
When ClickHouse exposes a source catalog through a [`DataLakeCatalog`](https://clickhouse.com/docs/engines/database-engines/datalake), map that source catalog to its ClickHouse database in the connection configuration:
476
+
477
+
```yaml
478
+
gateways:
479
+
clickhouse:
480
+
connection:
481
+
type: clickhouse
482
+
host: my-clickhouse-host
483
+
username: default
484
+
datalake_catalog_mapping:
485
+
lake: iceberg
486
+
```
487
+
488
+
SQLMesh renders `lake.analytics.orders` as `iceberg."analytics.orders"`. It combines the source
489
+
schema and table into one quoted table name because ClickHouse uses a two-level namespace. The
490
+
`iceberg`database must already exist and use the `DataLakeCatalog` engine. SQLMesh does not create
491
+
the database or manage its metastore or storage permissions.
492
+
493
+
For parsed queries that read data without writing the result to a ClickHouse table, SQLMesh rewrites
494
+
mapped source catalogs and passes other foreign catalog-qualified references through unchanged.
495
+
During rendering, SQLMesh strips the configured virtual catalog prefix, even before injection. For
496
+
`INSERT ... SELECT`and other query materializations, SQLMesh rewrites mapped sources and validates
497
+
the remaining catalog-qualified sources. With a non-empty mapping, each source catalog must appear
498
+
in the mapping. When SQLMesh has injected a virtual catalog, a source can also use that catalog.
499
+
SQLMesh raises an error for other catalogs. Without a non-empty mapping or an injected virtual
500
+
catalog, SQLMesh does not apply DataLake mapping or catalog validation during materialization.
501
+
502
+
This mapping rewrites parsed query sources. It does not rewrite table names used for metadata
503
+
operations such as `columns()` or `table_exists()`. The `sqlmesh create_external_models` command also
504
+
uses the source name as supplied when it fetches column types, so it cannot infer columns for a
505
+
source named `lake.analytics.orders` through the mapping. In a ClickHouse-only project, registering
506
+
that three-part logical name as an external model can also fail SQLMesh's schema nesting check,
507
+
because ClickHouse project models use two-part names.
508
+
509
+
To register column metadata until mapped names are supported there, use ClickHouse's physical
510
+
two-part name in both the external model and the query. For example, use
511
+
`iceberg."analytics.orders"`for the table exposed by database `iceberg`. This matches ClickHouse's
512
+
`database."schema.table"`name and keeps the model at ClickHouse's two-part nesting level. The
513
+
mapping still lets SQLMesh render queries that use mapped logical names. Without registered
514
+
column metadata, SQLMesh cannot fully optimize those queries or produce column-level lineage.
515
+
516
+
An injected virtual catalog takes precedence over a mapping key with the same name. SQLMesh treats
517
+
that name as its virtual catalog. Use different names for the virtual catalog and DataLakeCatalog
518
+
source catalogs. The mapping applies to query source references. It does not rewrite SQL strings
519
+
passed directly to the ClickHouse adapter or map `INSERT` targets.
520
+
521
+
View creation rejects a source that uses another catalog when a virtual catalog is injected, even
522
+
if the mapping includes that source. Mapped-source views work when no virtual catalog is injected.
523
+
473
524
## Local/Built-in Scheduler
474
525
475
526
**Engine Adapter Type**: `clickhouse`
@@ -498,3 +549,4 @@ With this configuration, ClickHouse models will appear as `ch_virtual.mydb.mytab
| `connection_pool_options` | Additional [options](https://clickhouse.com/docs/integrations/python#customizing-the-http-connection-pool) for the HTTP connection pool | dict | N |
500
551
| `virtual_catalog` | Override the virtual catalog name used when ClickHouse runs alongside a catalog-aware gateway (e.g. Trino). Defaults to `__{gateway_name}__`. See [Multi-gateway setup](#multi-gateway-setup) for details. | string | N |
552
+
| `datalake_catalog_mapping` | Map source catalogs to ClickHouse DataLakeCatalog databases. See [Reading catalog-backed physical tables](#reading-catalog-backed-physical-tables). | dict | N |
0 commit comments