Skip to content

Commit 3891d65

Browse files
mday-ionicksspirit
andauthored
fix(clickhouse): suppress ON CLUSTER for object DDL in a Replicated database (#6003)
Signed-off-by: mday-io <mdaytn@gmail.com> Signed-off-by: Michael Day <mdaytn@gmail.com> Co-authored-by: Nick Muoh <nickspirit3+github@gmail.com>
1 parent d60480e commit 3891d65

4 files changed

Lines changed: 577 additions & 19 deletions

File tree

‎docs/integrations/engines/clickhouse.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ In general, you must be connected to a ClickHouse server to execute commands. By
4949

5050
To associate an object with a cluster, DDL commands that create or modify it must include the text `ON CLUSTER [your cluster name]`.
5151

52-
If you provide a cluster name in your SQLMesh connection configuration, SQLMesh will automatically inject the `ON CLUSTER` statement into the DDL commands for all objects created while executing the project. We provide more information about clusters in SQLMesh [below](#cluster-specification).
52+
If you provide a cluster name in your SQLMesh connection configuration, SQLMesh automatically adds `ON CLUSTER` to object DDL, except for metadata DDL targeting `Replicated` databases, which replicate those changes themselves. We provide more information about clusters in SQLMesh [below](#cluster-specification).
5353

5454
#### ClickHouse Cloud mode
5555

@@ -96,7 +96,9 @@ For example, we could create a table `my_schema.my_table` on cluster `TheCluster
9696

9797
To create SQLMesh objects on a cluster, provide the cluster name to the `cluster` key in the SQLMesh connection definition (see all connection parameters [below](#localbuilt-in-scheduler)).
9898

99-
SQLMesh will automatically inject the `ON CLUSTER` clause and cluster name you provide into all project DDL statements.
99+
SQLMesh automatically adds the `ON CLUSTER` clause and cluster name you provide to object DDL, except for metadata DDL targeting `Replicated` databases. SQLMesh retains `ON CLUSTER` for database creation and removal and for partition operations.
100+
101+
Partition overwrites in multi-shard `Replicated` databases remain unvalidated. ClickHouse can replicate an ALTER containing both `REPLACE PARTITION` and `DROP PARTITION`, which can conflict with `ON CLUSTER`. This behavior predates the metadata DDL policy described above and depends on the ClickHouse version.
100102

101103
## Model definition
102104

‎sqlmesh/core/engine_adapter/base.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1043,12 +1043,15 @@ def _build_create_table_exp(
10431043
) -> exp.Create:
10441044
exists = False if replace else exists
10451045
catalog_name = None
1046+
target_table: t.Optional[exp.Table] = None
10461047
if not isinstance(table_name_or_schema, exp.Schema):
10471048
table_name_or_schema = exp.to_table(table_name_or_schema)
10481049
catalog_name = table_name_or_schema.catalog
1050+
target_table = table_name_or_schema
10491051
else:
10501052
if isinstance(table_name_or_schema.this, exp.Table):
10511053
catalog_name = table_name_or_schema.this.catalog
1054+
target_table = table_name_or_schema.this
10521055

10531056
properties = (
10541057
self._build_table_properties_exp(
@@ -1057,6 +1060,9 @@ def _build_create_table_exp(
10571060
target_columns_to_types=target_columns_to_types,
10581061
table_description=table_description,
10591062
table_kind=table_kind,
1063+
# Passed so an adapter can vary properties by target object, not just by
1064+
# connection. Additive: every override accepts **kwargs.
1065+
table=target_table,
10601066
)
10611067
if kwargs or table_description
10621068
else None
@@ -1364,6 +1370,8 @@ def create_view(
13641370
else None
13651371
),
13661372
physical_cluster=create_kwargs.pop("physical_cluster", None),
1373+
# See the note on the table-properties call site above.
1374+
table=exp.to_table(view_name),
13671375
)
13681376
if create_view_properties:
13691377
for view_property in create_view_properties.expressions:

0 commit comments

Comments
 (0)