Skip to content

Commit bcadfe1

Browse files
authored
Merge branch 'main' into mday/nifty-darwin-df5zc5
2 parents bc8ae23 + 3fea21e commit bcadfe1

22 files changed

Lines changed: 1666 additions & 54 deletions

File tree

‎.github/dependabot.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,30 @@ updates:
44
directory: '/'
55
schedule:
66
interval: 'weekly'
7+
groups:
8+
# Storybook add-ons declare a peer dependency on the matching `storybook`
9+
# release, so they have to move together. Bump them in a single PR.
10+
storybook:
11+
patterns:
12+
- 'storybook'
13+
- '@storybook/*'
14+
- '@chromatic-com/storybook'
15+
- 'eslint-plugin-storybook'
16+
update-types:
17+
- 'minor'
18+
- 'patch'
19+
ignore:
20+
# A Storybook major upgrade has to be done by hand as one coordinated
21+
# migration (e.g. @storybook/addon-vitest 10 requires vitest 4+), so don't
22+
# open per-package major-bump PRs, which break the Storybook build.
23+
- dependency-name: 'storybook'
24+
update-types: ['version-update:semver-major']
25+
- dependency-name: '@storybook/*'
26+
update-types: ['version-update:semver-major']
27+
- dependency-name: '@chromatic-com/storybook'
28+
update-types: ['version-update:semver-major']
29+
- dependency-name: 'eslint-plugin-storybook'
30+
update-types: ['version-update:semver-major']
731
- package-ecosystem: 'github-actions'
832
directory: '/'
933
schedule:

‎.github/workflows/pr.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,10 @@ jobs:
150150
. ./venv/Scripts/activate
151151
python.exe -m pip install --upgrade pip
152152
make install-dev
153+
- name: Install DuckLake extension
154+
run: |
155+
. ./venv/Scripts/activate
156+
python -c "import duckdb; duckdb.connect().install_extension('ducklake')"
153157
- name: Run fast unit tests
154158
run: |
155159
. ./venv/Scripts/activate
@@ -267,6 +271,7 @@ jobs:
267271
trino,
268272
spark,
269273
clickhouse,
274+
clickhouse-datalake,
270275
risingwave,
271276
starrocks,
272277
]
@@ -293,6 +298,9 @@ jobs:
293298
run: |
294299
source .venv/bin/activate
295300
make ${{ matrix.engine }}-test
301+
- name: Stop DataLake services
302+
if: ${{ always() && matrix.engine == 'clickhouse-datalake' }}
303+
run: make clickhouse-datalake-down
296304
- name: Upload test results
297305
uses: actions/upload-artifact@v7
298306
if: ${{ !cancelled() }}

‎Makefile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
.PHONY: docs
2+
.PHONY: clickhouse-datalake-up clickhouse-datalake-test clickhouse-datalake-down
23

34
ifdef UV
45
PIP := uv pip
@@ -192,6 +193,21 @@ engine-%-up: engine-%-install engine-docker-%-up
192193
engine-%-down:
193194
docker compose -f ./tests/core/engine_adapter/integration/docker/compose.${*}.yaml down -v
194195

196+
CLICKHOUSE_DATALAKE_COMPOSE = docker compose --project-name sqlmesh-clickhouse-datalake -f ./tests/core/engine_adapter/integration/docker/compose.clickhouse-datalake.yaml
197+
198+
clickhouse-datalake-up:
199+
$(CLICKHOUSE_DATALAKE_COMPOSE) up -d --wait --wait-timeout 120 clickhouse minio-init rest
200+
$(CLICKHOUSE_DATALAKE_COMPOSE) run --build --rm seed
201+
202+
clickhouse-datalake-test:
203+
docker info >/dev/null
204+
$(MAKE) engine-clickhouse-install
205+
$(MAKE) clickhouse-datalake-up
206+
SQLMESH_RUN_CLICKHOUSE_DATALAKE=1 pytest -n 0 -m "docker and clickhouse_datalake" tests/core/engine_adapter/integration/test_integration_clickhouse_datalake.py --junitxml=test-results/junit-clickhouse-datalake.xml
207+
208+
clickhouse-datalake-down:
209+
$(CLICKHOUSE_DATALAKE_COMPOSE) down -v --remove-orphans
210+
195211
##################
196212
# Docker Engines #
197213
##################

‎docs/concepts/audits.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -477,7 +477,7 @@ MODEL (
477477
#### valid_uuid, valid_uuid_non_blocking
478478
Ensures that all non-NULL rows of a column contain a string with the UUID structure.
479479

480-
UUID structure determined by matching regular expression `'^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'`.
480+
UUID structure determined by matching regular expression `'^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'`.
481481

482482
This example asserts that all `uuid` values have the UUID structure:
483483

‎docs/integrations/engines/clickhouse.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -470,6 +470,57 @@ gateways:
470470

471471
With this configuration, ClickHouse models will appear as `ch_virtual.mydb.mytable` in plan output instead of `__clickhouse__.mydb.mytable`.
472472

473+
### Reading catalog-backed physical tables
474+
475+
When ClickHouse exposes a source catalog through a [`DataLakeCatalog`](https://clickhouse.com/docs/engines/database-engines/datalake), map that source catalog to its ClickHouse database in the connection configuration:
476+
477+
```yaml
478+
gateways:
479+
clickhouse:
480+
connection:
481+
type: clickhouse
482+
host: my-clickhouse-host
483+
username: default
484+
datalake_catalog_mapping:
485+
lake: iceberg
486+
```
487+
488+
SQLMesh renders `lake.analytics.orders` as `iceberg."analytics.orders"`. It combines the source
489+
schema and table into one quoted table name because ClickHouse uses a two-level namespace. The
490+
`iceberg` database must already exist and use the `DataLakeCatalog` engine. SQLMesh does not create
491+
the database or manage its metastore or storage permissions.
492+
493+
For parsed queries that read data without writing the result to a ClickHouse table, SQLMesh rewrites
494+
mapped source catalogs and passes other foreign catalog-qualified references through unchanged.
495+
During rendering, SQLMesh strips the configured virtual catalog prefix, even before injection. For
496+
`INSERT ... SELECT` and other query materializations, SQLMesh rewrites mapped sources and validates
497+
the remaining catalog-qualified sources. With a non-empty mapping, each source catalog must appear
498+
in the mapping. When SQLMesh has injected a virtual catalog, a source can also use that catalog.
499+
SQLMesh raises an error for other catalogs. Without a non-empty mapping or an injected virtual
500+
catalog, SQLMesh does not apply DataLake mapping or catalog validation during materialization.
501+
502+
This mapping rewrites parsed query sources. It does not rewrite table names used for metadata
503+
operations such as `columns()` or `table_exists()`. The `sqlmesh create_external_models` command also
504+
uses the source name as supplied when it fetches column types, so it cannot infer columns for a
505+
source named `lake.analytics.orders` through the mapping. In a ClickHouse-only project, registering
506+
that three-part logical name as an external model can also fail SQLMesh's schema nesting check,
507+
because ClickHouse project models use two-part names.
508+
509+
To register column metadata until mapped names are supported there, use ClickHouse's physical
510+
two-part name in both the external model and the query. For example, use
511+
`iceberg."analytics.orders"` for the table exposed by database `iceberg`. This matches ClickHouse's
512+
`database."schema.table"` name and keeps the model at ClickHouse's two-part nesting level. The
513+
mapping still lets SQLMesh render queries that use mapped logical names. Without registered
514+
column metadata, SQLMesh cannot fully optimize those queries or produce column-level lineage.
515+
516+
An injected virtual catalog takes precedence over a mapping key with the same name. SQLMesh treats
517+
that name as its virtual catalog. Use different names for the virtual catalog and DataLakeCatalog
518+
source catalogs. The mapping applies to query source references. It does not rewrite SQL strings
519+
passed directly to the ClickHouse adapter or map `INSERT` targets.
520+
521+
View creation rejects a source that uses another catalog when a virtual catalog is injected, even
522+
if the mapping includes that source. Mapped-source views work when no virtual catalog is injected.
523+
473524
## Local/Built-in Scheduler
474525

475526
**Engine Adapter Type**: `clickhouse`
@@ -498,3 +549,4 @@ With this configuration, ClickHouse models will appear as `ch_virtual.mydb.mytab
498549
| `connection_settings` | Additional [connection settings](https://clickhouse.com/docs/integrations/python#settings-argument) | dict | N |
499550
| `connection_pool_options` | Additional [options](https://clickhouse.com/docs/integrations/python#customizing-the-http-connection-pool) for the HTTP connection pool | dict | N |
500551
| `virtual_catalog` | Override the virtual catalog name used when ClickHouse runs alongside a catalog-aware gateway (e.g. Trino). Defaults to `__{gateway_name}__`. See [Multi-gateway setup](#multi-gateway-setup) for details. | string | N |
552+
| `datalake_catalog_mapping` | Map source catalogs to ClickHouse DataLakeCatalog databases. See [Reading catalog-backed physical tables](#reading-catalog-backed-physical-tables). | dict | N |

‎pnpm-lock.yaml‎

Lines changed: 20 additions & 20 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,7 @@ markers = [
273273
"athena: test for Athena",
274274
"bigquery: test for BigQuery",
275275
"clickhouse: test for Clickhouse (standalone mode / cluster mode)",
276+
"clickhouse_datalake: test for ClickHouse DataLakeCatalog mapping",
276277
"clickhouse_cloud: test for Clickhouse (cloud mode)",
277278
"databricks: test for Databricks",
278279
"db2: test for Db2",

‎sqlmesh/core/audit/builtin.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,7 @@ def create_non_blocking_copy(audit: Audit) -> Audit:
297297
SELECT *
298298
FROM @this_model
299299
WHERE @AND(
300-
NOT REGEXP_LIKE(LOWER(@column), '^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'),
300+
NOT REGEXP_LIKE(LOWER(@column), '^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'),
301301
@condition,
302302
)
303303

‎sqlmesh/core/audit/definition.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,9 @@ def load_multiple_audits(
538538
yield load_audit(
539539
expressions=audit_block,
540540
path=path,
541+
module_path=module_path,
542+
macros=macros,
543+
jinja_macros=jinja_macros,
541544
dialect=dialect,
542545
default_catalog=default_catalog,
543546
variables=variables,

‎sqlmesh/core/config/connection.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2244,6 +2244,7 @@ class ClickhouseConnectionConfig(ConnectionConfig):
22442244
port: t.Optional[int] = None
22452245
cluster: t.Optional[str] = None
22462246
virtual_catalog: t.Optional[str] = None
2247+
datalake_catalog_mapping: t.Dict[str, str] = Field(default_factory=dict)
22472248
connect_timeout: int = 10
22482249
send_receive_timeout: int = 300
22492250
query_limit: int = 0
@@ -2291,6 +2292,21 @@ def validate_virtual_catalog(cls, v: t.Optional[str]) -> t.Optional[str]:
22912292
)
22922293
return v
22932294

2295+
@field_validator("datalake_catalog_mapping")
2296+
def validate_datalake_catalog_mapping(cls, v: t.Dict[str, str]) -> t.Dict[str, str]:
2297+
for source_catalog, target_database in v.items():
2298+
if not source_catalog.strip() or "." in source_catalog:
2299+
raise ConfigError(
2300+
"datalake_catalog_mapping keys must be non-empty single identifiers "
2301+
f"(got: {source_catalog!r})"
2302+
)
2303+
if not target_database.strip() or "." in target_database:
2304+
raise ConfigError(
2305+
"datalake_catalog_mapping values must be non-empty ClickHouse database "
2306+
f"identifiers (got: {target_database!r})"
2307+
)
2308+
return v
2309+
22942310
@property
22952311
def _connection_kwargs_keys(self) -> t.Set[str]:
22962312
kwargs = {
@@ -2357,6 +2373,7 @@ def _extra_engine_config(self) -> t.Dict[str, t.Any]:
23572373
"cluster": self.cluster,
23582374
"cloud_mode": self.cloud_mode,
23592375
"virtual_catalog": self.virtual_catalog,
2376+
"datalake_catalog_mapping": self.datalake_catalog_mapping,
23602377
}
23612378

23622379
@property

0 commit comments

Comments
 (0)