Skip to content

Commit b0ed0ca

Browse files
authored
Merge branch 'main' into fix/snowflake-iceberg-column-comments
2 parents 5ef3a92 + c4dbac4 commit b0ed0ca

108 files changed

Lines changed: 11175 additions & 1170 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/dependabot.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,30 @@ updates:
44
directory: '/'
55
schedule:
66
interval: 'weekly'
7+
groups:
8+
# Storybook add-ons declare a peer dependency on the matching `storybook`
9+
# release, so they have to move together. Bump them in a single PR.
10+
storybook:
11+
patterns:
12+
- 'storybook'
13+
- '@storybook/*'
14+
- '@chromatic-com/storybook'
15+
- 'eslint-plugin-storybook'
16+
update-types:
17+
- 'minor'
18+
- 'patch'
19+
ignore:
20+
# A Storybook major upgrade has to be done by hand as one coordinated
21+
# migration (e.g. @storybook/addon-vitest 10 requires vitest 4+), so don't
22+
# open per-package major-bump PRs, which break the Storybook build.
23+
- dependency-name: 'storybook'
24+
update-types: ['version-update:semver-major']
25+
- dependency-name: '@storybook/*'
26+
update-types: ['version-update:semver-major']
27+
- dependency-name: '@chromatic-com/storybook'
28+
update-types: ['version-update:semver-major']
29+
- dependency-name: 'eslint-plugin-storybook'
30+
update-types: ['version-update:semver-major']
731
- package-ecosystem: 'github-actions'
832
directory: '/'
933
schedule:

‎.github/scripts/install-prerequisites.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ ENGINE_DEPENDENCIES=""
1717

1818
if [ "$ENGINE" == "spark" ]; then
1919
ENGINE_DEPENDENCIES="default-jdk"
20+
elif [ "$ENGINE" == "db2" ]; then
21+
ENGINE_DEPENDENCIES="libxml2-dev build-essential"
2022
elif [ "$ENGINE" == "fabric" ]; then
2123
echo "Installing Microsoft package repository"
2224

‎.github/scripts/wait-for-db.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,20 @@ risingwave_ready() {
9090
probe_port 4566
9191
}
9292

93+
db2_ready() {
94+
probe_port 50001
95+
96+
echo "Waiting for Db2 to finish initialising (this can take 2-4 minutes)..."
97+
while true; do
98+
if docker exec db2 su - db2inst1 -c "db2 connect to TESTDB" > /dev/null 2>&1; then
99+
echo "Db2 is accepting connections"
100+
break
101+
fi
102+
echo "Db2 not yet ready; sleeping 15s..."
103+
sleep 15
104+
done
105+
}
106+
93107
echo "Waiting for $ENGINE to be ready..."
94108

95109
READINESS_FUNC="${ENGINE}_ready"

‎.github/workflows/db2.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: db2
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- 'sqlmesh/core/engine_adapter/db2.py'
9+
- 'tests/core/engine_adapter/test_db2.py'
10+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
11+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
12+
- '.github/workflows/db2.yaml'
13+
- 'pyproject.toml'
14+
pull_request:
15+
branches:
16+
- main
17+
paths:
18+
- 'sqlmesh/core/engine_adapter/db2.py'
19+
- 'tests/core/engine_adapter/test_db2.py'
20+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
21+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
22+
- '.github/workflows/db2.yaml'
23+
- 'pyproject.toml'
24+
25+
concurrency:
26+
group: db2-${{ github.event.pull_request.number || github.sha }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read
31+
32+
jobs:
33+
db2-test:
34+
runs-on: ubuntu-latest
35+
timeout-minutes: 25
36+
env:
37+
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
38+
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
39+
UV: '1'
40+
steps:
41+
- uses: actions/checkout@v7
42+
- name: Set up Python
43+
uses: actions/setup-python@v7
44+
with:
45+
python-version: '3.12'
46+
- name: Install uv
47+
uses: astral-sh/setup-uv@v7
48+
- name: Install SQLMesh dev dependencies
49+
run: |
50+
uv venv .venv
51+
source .venv/bin/activate
52+
make install-dev
53+
- name: Install OS-level dependencies
54+
run: ./.github/scripts/install-prerequisites.sh "db2"
55+
- name: Run tests
56+
run: |
57+
source .venv/bin/activate
58+
make db2-test
59+
- name: Upload test results
60+
uses: actions/upload-artifact@v7
61+
if: ${{ !cancelled() }}
62+
with:
63+
name: test-results-docker-db2
64+
path: test-results/
65+
retention-days: 7

‎.github/workflows/pr.yaml‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ jobs:
1717
python: ${{ steps.filter.outputs.python }}
1818
client: ${{ steps.filter.outputs.client }}
1919
vscode: ${{ steps.filter.outputs.vscode }}
20+
lsp: ${{ steps.filter.outputs.lsp }}
2021
ci: ${{ steps.filter.outputs.ci }}
2122
steps:
2223
- uses: actions/checkout@v7
@@ -37,6 +38,8 @@ jobs:
3738
- 'web/client/**'
3839
vscode:
3940
- 'vscode/**'
41+
lsp:
42+
- 'sqlmesh/lsp/**'
4043
ci:
4144
- '.github/**'
4245
- 'Makefile'
@@ -264,6 +267,7 @@ jobs:
264267
trino,
265268
spark,
266269
clickhouse,
270+
clickhouse-datalake,
267271
risingwave,
268272
starrocks,
269273
]
@@ -290,6 +294,9 @@ jobs:
290294
run: |
291295
source .venv/bin/activate
292296
make ${{ matrix.engine }}-test
297+
- name: Stop DataLake services
298+
if: ${{ always() && matrix.engine == 'clickhouse-datalake' }}
299+
run: make clickhouse-datalake-down
293300
- name: Upload test results
294301
uses: actions/upload-artifact@v7
295302
if: ${{ !cancelled() }}
@@ -427,10 +434,12 @@ jobs:
427434
- name: Run CI
428435
run: pnpm run ci
429436
test-vscode-e2e:
430-
runs-on:
431-
labels: [ubuntu-2204-8]
432-
# As at 2026-01-12 this job flakes 100% of the time. It needs investigation
433-
if: false
437+
needs: changes
438+
if:
439+
needs.changes.outputs.vscode == 'true' || needs.changes.outputs.lsp ==
440+
'true' || needs.changes.outputs.ci == 'true' || github.ref ==
441+
'refs/heads/main'
442+
runs-on: ubuntu-latest
434443
steps:
435444
- uses: actions/checkout@v7
436445
- uses: actions/setup-node@v7
@@ -453,7 +462,9 @@ jobs:
453462
source .venv/bin/activate
454463
make install-dev
455464
- name: Install code-server
456-
run: curl -fsSL https://code-server.dev/install.sh | sh
465+
# Pinned so that a code-server release cannot change the editor UI the
466+
# tests drive without the change showing up here first.
467+
run: curl -fsSL https://code-server.dev/install.sh | sh -s -- --version 4.137.0
457468
- name: Install Playwright browsers
458469
working-directory: ./vscode/extension
459470
run: pnpm exec playwright install
@@ -470,6 +481,10 @@ jobs:
470481
path: vscode/extension/playwright-report/
471482
retention-days: 30
472483
test-dbt-versions:
484+
needs: changes
485+
if:
486+
needs.changes.outputs.python == 'true' || needs.changes.outputs.ci ==
487+
'true' || github.ref == 'refs/heads/main'
473488
runs-on: ubuntu-latest
474489
strategy:
475490
fail-fast: false
@@ -554,7 +569,8 @@ jobs:
554569
run: |
555570
source ../../.venv/bin/activate
556571
sed -i 's/target: in_memory/target: postgres/g' profiles.yml
557-
if [[ $(echo -e "${{ matrix.dbt-version }}\n1.5.0" | sort -V | head -n1) == "${{ matrix.dbt-version }}" ]] && [[ "${{ matrix.dbt-version }}" != "1.5.0" ]]; then
572+
if [[ "${{ matrix.dbt-version }}" == "1.3" ]] || \
573+
[[ "${{ matrix.dbt-version }}" == "1.4" ]]; then
558574
echo "DBT version is ${{ matrix.dbt-version }} (< 1.5.0), removing version parameters..."
559575
sed -i -e 's/, version=1) }}/) }}/g' -e 's/, v=1) }}/) }}/g' models/top_waiters.sql
560576
else

‎.prettierignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ docs
3030
examples
3131
posts
3232
.circleci
33+
.github/
3334
README.md
3435
mkdocs.yml
3536
.readthedocs.yaml

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,12 @@ See [docs/development.md](docs/development.md) for full setup instructions. Key
6565
python -m venv .venv
6666
source .venv/bin/activate
6767
make install-dev
68-
make style # Run before submitting
68+
make style # Run before submitting
6969
make fast-test # Quick test suite
7070
```
7171

72+
Optionally, `make install-pre-commit` installs git hooks so ruff and mypy run on `git commit`. Hooks do not replace `make style`: they run on staged files, while CI runs `make style` across the tree.
73+
7274
## Coding Standards
7375

7476
- Run `make style` before submitting a pull request

‎Makefile‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
.PHONY: docs
2+
.PHONY: clickhouse-datalake-up clickhouse-datalake-test clickhouse-datalake-down
23

34
ifdef UV
45
PIP := uv pip
@@ -132,13 +133,13 @@ engine-up: engine-clickhouse-up engine-mssql-up engine-mysql-up engine-postgres-
132133
engine-down: engine-clickhouse-down engine-mssql-down engine-mysql-down engine-postgres-down engine-spark-down engine-trino-down
133134

134135
fast-test:
135-
pytest -n auto -m "fast and not cicdonly and not isolated" --junitxml=test-results/junit-fast-test.xml && pytest -m "isolated and not slow" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
136+
pytest -n auto -m "fast and not cicdonly and not isolated" --junitxml=test-results/junit-fast-test.xml && pytest -m "isolated and not slow and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and fork_isolated and not slow" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and fork_isolated and not slow" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
136137

137138
slow-test:
138-
pytest -n auto -m "(fast or slow) and not cicdonly and not isolated" && pytest -m "isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
139+
pytest -n auto -m "(fast or slow) and not cicdonly and not isolated" && pytest -m "isolated and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and fork_isolated" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and fork_isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
139140

140141
cicd-test:
141-
pytest -n auto -m "(fast or slow) and not pyspark and not isolated" --junitxml=test-results/junit-cicd.xml && pytest -m "pyspark" && pytest -m "isolated and not pyspark" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
142+
pytest -n auto -m "(fast or slow) and not pyspark and not isolated" --junitxml=test-results/junit-cicd.xml && pytest -m "pyspark" && pytest -m "isolated and not pyspark and not fork_isolated" && pytest tests/test_forking.py::test_parallel_load -m "isolated and not pyspark and fork_isolated" && pytest tests/test_forking.py::test_parallel_load_multi_repo -m "isolated and not pyspark and fork_isolated" && pytest -m "registry_isolation" && pytest -m "dialect_isolated"
142143

143144
core-fast-test:
144145
pytest -n auto -m "fast and not web and not github and not dbt and not jupyter"
@@ -192,6 +193,21 @@ engine-%-up: engine-%-install engine-docker-%-up
192193
engine-%-down:
193194
docker compose -f ./tests/core/engine_adapter/integration/docker/compose.${*}.yaml down -v
194195

196+
CLICKHOUSE_DATALAKE_COMPOSE = docker compose --project-name sqlmesh-clickhouse-datalake -f ./tests/core/engine_adapter/integration/docker/compose.clickhouse-datalake.yaml
197+
198+
clickhouse-datalake-up:
199+
$(CLICKHOUSE_DATALAKE_COMPOSE) up -d --wait --wait-timeout 120 clickhouse minio-init rest
200+
$(CLICKHOUSE_DATALAKE_COMPOSE) run --build --rm seed
201+
202+
clickhouse-datalake-test:
203+
docker info >/dev/null
204+
$(MAKE) engine-clickhouse-install
205+
$(MAKE) clickhouse-datalake-up
206+
SQLMESH_RUN_CLICKHOUSE_DATALAKE=1 pytest -n 0 -m "docker and clickhouse_datalake" tests/core/engine_adapter/integration/test_integration_clickhouse_datalake.py --junitxml=test-results/junit-clickhouse-datalake.xml
207+
208+
clickhouse-datalake-down:
209+
$(CLICKHOUSE_DATALAKE_COMPOSE) down -v --remove-orphans
210+
195211
##################
196212
# Docker Engines #
197213
##################
@@ -222,6 +238,9 @@ risingwave-test: engine-risingwave-up
222238

223239
starrocks-test: engine-starrocks-up
224240
pytest -n auto -m "starrocks" --reruns 3 --junitxml=test-results/junit-starrocks.xml
241+
242+
db2-test: engine-db2-up
243+
pytest -n auto -m "db2" --reruns 3 --junitxml=test-results/junit-db2.xml
225244

226245
#################
227246
# Cloud Engines #
@@ -235,7 +254,10 @@ bigquery-test: guard-BIGQUERY_KEYFILE engine-bigquery-install
235254
pytest -n auto -m "bigquery" --reruns 3 --junitxml=test-results/junit-bigquery.xml
236255

237256
databricks-test: guard-DATABRICKS_CATALOG guard-DATABRICKS_SERVER_HOSTNAME guard-DATABRICKS_HTTP_PATH guard-DATABRICKS_CONNECT_VERSION engine-databricks-install
238-
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}'
257+
# Connect 15.3.x requires numpy<2 and pyarrow>=4 (no upper bound). Cap PyArrow
258+
# in the same resolve that downgrades NumPy, or PyArrow 25+ is left installed
259+
# and cannot be imported.
260+
$(PIP) install 'databricks-connect==${DATABRICKS_CONNECT_VERSION}' 'pyarrow<25'
239261
pytest -n auto -m "databricks" --reruns 3 --junitxml=test-results/junit-databricks.xml
240262

241263
redshift-test: guard-REDSHIFT_HOST guard-REDSHIFT_USER guard-REDSHIFT_PASSWORD guard-REDSHIFT_DATABASE engine-redshift-install

‎docs/concepts/audits.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -477,7 +477,7 @@ MODEL (
477477
#### valid_uuid, valid_uuid_non_blocking
478478
Ensures that all non-NULL rows of a column contain a string with the UUID structure.
479479

480-
UUID structure determined by matching regular expression `'^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'`.
480+
UUID structure determined by matching regular expression `'^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$'`.
481481

482482
This example asserts that all `uuid` values have the UUID structure:
483483

‎docs/concepts/tests.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -463,6 +463,37 @@ You can also run tests that match a pattern or substring using a glob pathname e
463463
$ sqlmesh test tests/test_*
464464
```
465465

466+
Passing the path of a model file runs the tests for that model, which is useful for commit hooks and other tools that work with changed files rather than test names:
467+
468+
```
469+
$ sqlmesh test models/full_model.sql
470+
```
471+
472+
Model files and test files can be mixed, and the results are unioned. A test selected by more than one argument still runs only once, so the following runs each of `full_model`'s tests a single time even though both arguments cover them:
473+
474+
```
475+
$ sqlmesh test models/full_model.sql tests/test_full_model.yaml
476+
```
477+
478+
An argument that is neither a known model file nor a known test file is an error, so a mistyped or stale path fails instead of quietly running no tests. A model that simply has no tests is not an error.
479+
480+
You can pass `--local` to run tests without loading state from the configured state connection:
481+
482+
``` bash
483+
$ sqlmesh test --local
484+
```
485+
486+
This keeps offline runs and commit hooks from opening a connection to the state backend.
487+
488+
In multi-repository setups, or when running tests for only a subset of projects, models that exist only in remote state are not loaded under `--local`. Unlike [`sqlmesh lint --local`](../guides/linter.md), which reports additional errors in that situation, a test whose model is missing is **skipped with a warning and the run still succeeds**:
489+
490+
```
491+
[WARNING] Model '"memory"."bronze"."a"' was not found at tests/test_a.yaml
492+
.**Successfully Ran `1` Tests Against `duckdb`**
493+
```
494+
495+
So a passing exit code alone does not mean every test you expected actually ran. Watch the output for these warnings, and keep in mind that a hook using `--local` will not fail on them.
496+
466497
### Testing using notebooks
467498

468499
You can execute tests on demand using the `%run_test` notebook magic as follows:

0 commit comments

Comments
 (0)