Skip to content

Commit b35567e

Browse files
authored
Merge branch 'main' into fix/audit-at-least-one-empty-table
2 parents 735b26f + 8c71de8 commit b35567e

11 files changed

Lines changed: 1195 additions & 21 deletions

File tree

‎.github/workflows/release.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,13 @@ jobs:
4040
- name: Set up Python
4141
uses: actions/setup-python@v7
4242
with:
43-
python-version: '3.10'
43+
# 3.13.17+ bundles pip 26.2.2+, which no longer trusts the compromised eMudhra/emSign root CA.
44+
python-version: '3.13'
4445
- name: Install uv
4546
uses: astral-sh/setup-uv@v7
47+
- name: Check pip includes the eMudhra/emSign CA removal
48+
run: |
49+
python -c "import sys, pip; v = tuple(int(p) for p in pip.__version__.split('.')[:3]); sys.exit(0 if v >= (26, 2, 2) else f'pip {pip.__version__} predates the eMudhra/emSign CA removal; need >= 26.2.2')"
4650
- name: Install build dependencies
4751
run: pip install build twine setuptools_scm
4852
- name: Publish Python package

‎docs/guides/configuration.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1213,10 +1213,12 @@ Example configuration:
12131213

12141214
#### Default connections/scheduler
12151215

1216-
The `default_connection`, `default_test_connection`, and `default_scheduler` keys are used to specify shared defaults across multiple gateways.
1216+
The `default_connection`, `default_test_connection`, and `default_scheduler` keys specify a connection or scheduler for gateways that do not define their own.
12171217

12181218
For example, you might have a specific connection where your tests should run regardless of which gateway is being used. Instead of duplicating the test connection information in each gateway specification, specify it once in the `default_test_connection` key.
12191219

1220+
A default is only used when a gateway omits the corresponding key entirely. If a gateway specifies its own `connection`, `test_connection`, or `scheduler`, SQLMesh uses that configuration as written and ignores the default. Fields are not merged, so a gateway connection cannot inherit individual fields such as `type` from `default_connection`. Each gateway connection must be a complete connection configuration.
1221+
12201222
Example configuration specifying a Postgres default connection, in-memory DuckDB default test connection, and builtin default scheduler:
12211223

12221224
=== "YAML"

‎docs/reference/configuration.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -264,13 +264,15 @@ If a configuration contains multiple gateways, SQLMesh will use the first one in
264264

265265
### Default connections/scheduler
266266

267-
The `default_connection`, `default_test_connection`, and `default_scheduler` keys are used to specify shared defaults across multiple gateways.
267+
The `default_connection`, `default_test_connection`, and `default_scheduler` keys specify a connection or scheduler for gateways that do not define their own.
268268

269269
For example, you might have a specific connection where your tests should run regardless of which gateway is being used. Instead of duplicating the test connection information in each gateway specification, specify it once in the `default_test_connection` key.
270270

271+
A default is only used when a gateway omits the corresponding key entirely. If a gateway specifies its own `connection`, `test_connection`, or `scheduler`, SQLMesh uses that configuration as written and ignores the default. Fields are not merged, so a gateway connection cannot inherit individual fields such as `type` from `default_connection`.
272+
271273
| Option | Description | Type | Required |
272274
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------: | :------: |
273-
| `default_connection` | The default connection to use if one is not specified in a gateway (Default: A DuckDB connection that creates an in-memory database) | connection | N |
275+
| `default_connection` | The connection to use for any gateway that does not specify a `connection` | connection | N |
274276
| `default_test_connection` | The default connection to use when running tests if one is not specified in a gateway (Default: A DuckDB connection that creates an in-memory database) | connection | N |
275277
| `default_scheduler` | The default scheduler configuration to use if one is not specified in a gateway (Default: built-in scheduler) | scheduler | N |
276278

‎sqlmesh/core/context.py‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -420,6 +420,7 @@ def __init__(
420420
self._engine_adapter: t.Optional[EngineAdapter] = None
421421
self._linters: t.Dict[str, Linter] = {}
422422
self._loaded: bool = False
423+
self._materializations_loaded: bool = False
423424
self._load_state: bool = load_state
424425
self._uncached_model_names: t.Set[str] = set()
425426
self._selector_cls = selector or NativeSelector
@@ -807,6 +808,7 @@ def load(
807808
)
808809

809810
self._loaded = True
811+
self._materializations_loaded = True
810812
return self
811813

812814
def _update_model_schemas_and_validate(self, model_fqns: t.Optional[t.Set[str]] = None) -> None:
@@ -1017,6 +1019,10 @@ def run_janitor(
10171019

10181020
@python_api_analytics
10191021
def destroy(self) -> bool:
1022+
# Load first so snapshot unpickling cannot import a custom-kind module that this load then reimports.
1023+
# If loading fails, abort before invalidating any environments.
1024+
self._load_materializations()
1025+
10201026
success = False
10211027

10221028
# Collect resources to be deleted
@@ -3182,6 +3188,8 @@ def _run_janitor(
31823188
current_ts = now_timestamp()
31833189
failures: t.List[str] = []
31843190

3191+
self._load_materializations()
3192+
31853193
# Clean up expired environments by removing their views and schemas
31863194
failures.extend(
31873195
self._cleanup_environments(
@@ -3373,9 +3381,10 @@ def _register_notification_targets(self) -> None:
33733381
)
33743382

33753383
def _load_materializations(self) -> None:
3376-
if not self._loaded:
3384+
if not self._materializations_loaded:
33773385
for loader in self._loaders:
33783386
loader.load_materializations()
3387+
self._materializations_loaded = True
33793388

33803389
def _select_models_for_run(
33813390
self,

‎sqlmesh/core/engine_adapter/mixins.py‎

Lines changed: 43 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818

1919
if t.TYPE_CHECKING:
2020
from sqlmesh.core._typing import TableName
21+
from sqlmesh.core.engine_adapter.shared import DataObject
2122
from sqlmesh.core.engine_adapter._typing import (
2223
DCL,
2324
DF,
@@ -383,23 +384,53 @@ def get_alter_operations(
383384
self.get_data_objects(target_table_schema, {target_table.name}), 0
384385
)
385386

387+
clustering_operation: t.Optional[TableAlterClusterByOperation] = None
386388
if current_table_info and target_table_info:
387-
if target_table_info.is_clustered:
388-
if target_table_info.clustering_key and (
389-
current_table_info.clustering_key != target_table_info.clustering_key
390-
):
391-
operations.append(
392-
TableAlterChangeClusterKeyOperation(
393-
target_table=current_table,
394-
clustering_key=target_table_info.clustering_key,
395-
dialect=self.dialect,
396-
)
397-
)
389+
if target_table_info.is_clustered and target_table_info.clustering_key:
390+
# Include an unchanged key. Snowflake may have to drop it and set it
391+
# again when a key column is dropped and re-added. Other engines ignore
392+
# that operation in _apply_clustering_operation.
393+
clustering_operation = TableAlterChangeClusterKeyOperation(
394+
target_table=current_table,
395+
clustering_key=target_table_info.clustering_key,
396+
dialect=self.dialect,
397+
)
398398
elif current_table_info.is_clustered:
399-
operations.append(TableAlterDropClusterKeyOperation(target_table=current_table))
399+
clustering_operation = TableAlterDropClusterKeyOperation(target_table=current_table)
400+
401+
if clustering_operation:
402+
self._apply_clustering_operation(operations, clustering_operation, current_table_info)
400403

401404
return operations
402405

406+
def _clustering_key_is_unchanged(
407+
self,
408+
clustering_operation: "TableAlterClusterByOperation",
409+
current_table_info: t.Optional["DataObject"],
410+
) -> bool:
411+
return (
412+
isinstance(clustering_operation, TableAlterChangeClusterKeyOperation)
413+
and current_table_info is not None
414+
and current_table_info.clustering_key == clustering_operation.clustering_key
415+
)
416+
417+
def _apply_clustering_operation(
418+
self,
419+
operations: t.List[TableAlterOperation],
420+
clustering_operation: "TableAlterClusterByOperation",
421+
current_table_info: t.Optional["DataObject"],
422+
) -> None:
423+
"""Insert the clustering operation into the alter operations list.
424+
425+
The default implementation appends a real clustering change after all column
426+
changes. An unchanged key is ignored here. Snowflake overrides this so it can
427+
drop that key before removing a column the key names, then set the same key
428+
after the column comes back.
429+
"""
430+
if self._clustering_key_is_unchanged(clustering_operation, current_table_info):
431+
return
432+
operations.append(clustering_operation)
433+
403434

404435
def logical_merge(
405436
engine_adapter: EngineAdapter,

‎sqlmesh/core/engine_adapter/snowflake.py‎

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from __future__ import annotations
22

33
import contextlib
4+
import itertools
45
import logging
56
import typing as t
67

@@ -16,15 +17,18 @@
1617
ClusteredByMixin,
1718
RowDiffMixin,
1819
GrantsFromInfoSchemaMixin,
20+
TableAlterChangeClusterKeyOperation,
21+
TableAlterClusterByOperation,
22+
TableAlterDropClusterKeyOperation,
1923
)
24+
from sqlmesh.core.schema_diff import TableAlterDropColumnOperation, TableAlterOperation
2025
from sqlmesh.core.engine_adapter.shared import (
2126
CatalogSupport,
2227
DataObject,
2328
DataObjectType,
2429
SourceQuery,
2530
set_catalog,
2631
)
27-
from sqlmesh.core.schema_diff import TableAlterOperation
2832
from sqlmesh.utils import optional_import, get_source_columns_to_types
2933
from sqlmesh.utils.errors import SQLMeshError
3034
from sqlmesh.utils.pandas import columns_to_types_from_dtypes
@@ -692,6 +696,62 @@ def clone_table(
692696
**kwargs,
693697
)
694698

699+
def _apply_clustering_operation(
700+
self,
701+
operations: t.List[TableAlterOperation],
702+
clustering_operation: TableAlterClusterByOperation,
703+
current_table_info: t.Optional[DataObject],
704+
) -> None:
705+
"""Apply clustering DDL in the order Snowflake requires.
706+
707+
DROP COLUMN fails while the column is still in the clustering key, including
708+
inside an expression. CLUSTER BY fails on a column that does not exist yet. A
709+
type change that becomes DROP plus ADD still needs the key dropped first even
710+
when the key text is unchanged. An unchanged key with no matching column drop
711+
emits nothing. DESCRIBE names are compared as stored.
712+
"""
713+
clustering_key = current_table_info.clustering_key if current_table_info else None
714+
operations_include_drop_column = any(
715+
isinstance(op, TableAlterDropColumnOperation) for op in operations
716+
)
717+
718+
if clustering_key and operations_include_drop_column:
719+
referenced = self._clustering_key_column_names(clustering_key)
720+
drop_index = next(
721+
(
722+
i
723+
for i, op in enumerate(operations)
724+
if isinstance(op, TableAlterDropColumnOperation)
725+
and op.column_parts
726+
and op.column_parts[0].name in referenced
727+
),
728+
None,
729+
)
730+
if drop_index is not None:
731+
drop_key_op = TableAlterDropClusterKeyOperation(
732+
target_table=clustering_operation.target_table
733+
)
734+
operations.insert(drop_index, drop_key_op)
735+
if isinstance(clustering_operation, TableAlterChangeClusterKeyOperation):
736+
operations.append(clustering_operation)
737+
return
738+
if self._clustering_key_is_unchanged(clustering_operation, current_table_info):
739+
return
740+
operations.append(clustering_operation)
741+
742+
def _clustering_key_column_names(self, clustering_key: str) -> t.Set[str]:
743+
expressions = TableAlterChangeClusterKeyOperation(
744+
target_table=exp.to_table("_"),
745+
clustering_key=clustering_key,
746+
dialect=self.dialect,
747+
).cluster_key_expressions
748+
cluster_columns = itertools.chain.from_iterable(
749+
expression.find_all(exp.Column) for expression in expressions
750+
)
751+
return {
752+
normalize_identifiers(column, dialect=self.dialect).name for column in cluster_columns
753+
}
754+
695755
def alter_table(
696756
self,
697757
alter_expressions: t.Union[t.List[exp.Alter], t.List[TableAlterOperation]],

0 commit comments

Comments
 (0)