Skip to content

Commit beb2937

Browse files
authored
Merge branch 'main' into fix/fabric-ci
2 parents 7b5090d + d69c262 commit beb2937

78 files changed

Lines changed: 6198 additions & 622 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.devcontainer/devcontainer.json‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,7 @@
1111
"postCreateCommand": "bash .devcontainer/post-create-command.sh",
1212
"customizations": {
1313
"vscode": {
14-
"extensions": [
15-
"ms-python.python",
16-
"ms-python.vscode-pylance"
17-
]
14+
"extensions": ["ms-python.python", "ms-python.vscode-pylance"]
1815
}
1916
},
2017
"remoteUser": "vscode"

‎.github/scripts/install-prerequisites.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ ENGINE_DEPENDENCIES=""
1717

1818
if [ "$ENGINE" == "spark" ]; then
1919
ENGINE_DEPENDENCIES="default-jdk"
20+
elif [ "$ENGINE" == "db2" ]; then
21+
ENGINE_DEPENDENCIES="libxml2-dev build-essential"
2022
elif [ "$ENGINE" == "fabric" ]; then
2123
echo "Installing Microsoft package repository"
2224

‎.github/scripts/wait-for-db.sh‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,20 @@ risingwave_ready() {
9090
probe_port 4566
9191
}
9292

93+
db2_ready() {
94+
probe_port 50001
95+
96+
echo "Waiting for Db2 to finish initialising (this can take 2-4 minutes)..."
97+
while true; do
98+
if docker exec db2 su - db2inst1 -c "db2 connect to TESTDB" > /dev/null 2>&1; then
99+
echo "Db2 is accepting connections"
100+
break
101+
fi
102+
echo "Db2 not yet ready; sleeping 15s..."
103+
sleep 15
104+
done
105+
}
106+
93107
echo "Waiting for $ENGINE to be ready..."
94108

95109
READINESS_FUNC="${ENGINE}_ready"

‎.github/workflows/db2.yaml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: db2
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- 'sqlmesh/core/engine_adapter/db2.py'
9+
- 'tests/core/engine_adapter/test_db2.py'
10+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
11+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
12+
- '.github/workflows/db2.yaml'
13+
- 'pyproject.toml'
14+
pull_request:
15+
branches:
16+
- main
17+
paths:
18+
- 'sqlmesh/core/engine_adapter/db2.py'
19+
- 'tests/core/engine_adapter/test_db2.py'
20+
- 'tests/core/engine_adapter/integration/test_integration_db2.py'
21+
- 'tests/core/engine_adapter/integration/docker/compose.db2.yaml'
22+
- '.github/workflows/db2.yaml'
23+
- 'pyproject.toml'
24+
25+
concurrency:
26+
group: db2-${{ github.event.pull_request.number || github.sha }}
27+
cancel-in-progress: true
28+
29+
permissions:
30+
contents: read
31+
32+
jobs:
33+
db2-test:
34+
runs-on: ubuntu-latest
35+
timeout-minutes: 25
36+
env:
37+
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
38+
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
39+
UV: '1'
40+
steps:
41+
- uses: actions/checkout@v7
42+
- name: Set up Python
43+
uses: actions/setup-python@v7
44+
with:
45+
python-version: '3.12'
46+
- name: Install uv
47+
uses: astral-sh/setup-uv@v7
48+
- name: Install SQLMesh dev dependencies
49+
run: |
50+
uv venv .venv
51+
source .venv/bin/activate
52+
make install-dev
53+
- name: Install OS-level dependencies
54+
run: ./.github/scripts/install-prerequisites.sh "db2"
55+
- name: Run tests
56+
run: |
57+
source .venv/bin/activate
58+
make db2-test
59+
- name: Upload test results
60+
uses: actions/upload-artifact@v7
61+
if: ${{ !cancelled() }}
62+
with:
63+
name: test-results-docker-db2
64+
path: test-results/
65+
retention-days: 7

‎.github/workflows/pr.yaml‎

Lines changed: 41 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ jobs:
1616
outputs:
1717
python: ${{ steps.filter.outputs.python }}
1818
client: ${{ steps.filter.outputs.client }}
19+
vscode: ${{ steps.filter.outputs.vscode }}
1920
ci: ${{ steps.filter.outputs.ci }}
2021
steps:
2122
- uses: actions/checkout@v7
@@ -34,6 +35,8 @@ jobs:
3435
- 'pyproject.toml'
3536
client:
3637
- 'web/client/**'
38+
vscode:
39+
- 'vscode/**'
3740
ci:
3841
- '.github/**'
3942
- 'Makefile'
@@ -188,9 +191,10 @@ jobs:
188191

189192
ui-style:
190193
needs: [changes]
191-
if: false
192-
# needs.changes.outputs.client == 'true' || needs.changes.outputs.ci ==
193-
# 'true' || github.ref == 'refs/heads/main'
194+
if:
195+
needs.changes.outputs.client == 'true' || needs.changes.outputs.vscode ==
196+
'true' || needs.changes.outputs.ci == 'true' || github.ref ==
197+
'refs/heads/main'
194198
runs-on: ubuntu-latest
195199
steps:
196200
- uses: actions/checkout@v7
@@ -252,7 +256,17 @@ jobs:
252256
fail-fast: false
253257
matrix:
254258
engine:
255-
[duckdb, postgres, mysql, mssql, trino, spark, clickhouse, risingwave, starrocks]
259+
[
260+
duckdb,
261+
postgres,
262+
mysql,
263+
mssql,
264+
trino,
265+
spark,
266+
clickhouse,
267+
risingwave,
268+
starrocks,
269+
]
256270
env:
257271
PYTEST_XDIST_AUTO_NUM_WORKERS: 2
258272
SQLMESH__DISABLE_ANONYMIZED_ANALYTICS: '1'
@@ -393,10 +407,13 @@ jobs:
393407
retention-days: 7
394408

395409
test-vscode:
410+
needs: changes
411+
if:
412+
needs.changes.outputs.vscode == 'true' || needs.changes.outputs.ci ==
413+
'true' || github.ref == 'refs/heads/main'
396414
env:
397415
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
398416
runs-on: ubuntu-latest
399-
if: false
400417
steps:
401418
- uses: actions/checkout@v7
402419
- uses: actions/setup-node@v7
@@ -453,11 +470,27 @@ jobs:
453470
path: vscode/extension/playwright-report/
454471
retention-days: 30
455472
test-dbt-versions:
473+
needs: changes
474+
if:
475+
needs.changes.outputs.python == 'true' || needs.changes.outputs.ci ==
476+
'true' || github.ref == 'refs/heads/main'
456477
runs-on: ubuntu-latest
457478
strategy:
458479
fail-fast: false
459480
matrix:
460-
dbt-version: ['1.3', '1.4', '1.5', '1.6', '1.7', '1.8', '1.9', '1.10', '1.11', '1.12']
481+
dbt-version:
482+
[
483+
'1.3',
484+
'1.4',
485+
'1.5',
486+
'1.6',
487+
'1.7',
488+
'1.8',
489+
'1.9',
490+
'1.10',
491+
'1.11',
492+
'1.12',
493+
]
461494
steps:
462495
- uses: actions/checkout@v7
463496
- name: Set up Python
@@ -525,7 +558,8 @@ jobs:
525558
run: |
526559
source ../../.venv/bin/activate
527560
sed -i 's/target: in_memory/target: postgres/g' profiles.yml
528-
if [[ $(echo -e "${{ matrix.dbt-version }}\n1.5.0" | sort -V | head -n1) == "${{ matrix.dbt-version }}" ]] && [[ "${{ matrix.dbt-version }}" != "1.5.0" ]]; then
561+
if [[ "${{ matrix.dbt-version }}" == "1.3" ]] || \
562+
[[ "${{ matrix.dbt-version }}" == "1.4" ]]; then
529563
echo "DBT version is ${{ matrix.dbt-version }} (< 1.5.0), removing version parameters..."
530564
sed -i -e 's/, version=1) }}/) }}/g' -e 's/, v=1) }}/) }}/g' models/top_waiters.sql
531565
else

‎.prettierignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ docs
3030
examples
3131
posts
3232
.circleci
33+
.github/
3334
README.md
3435
mkdocs.yml
3536
.readthedocs.yaml

‎CONTRIBUTING.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,12 @@ See [docs/development.md](docs/development.md) for full setup instructions. Key
6565
python -m venv .venv
6666
source .venv/bin/activate
6767
make install-dev
68-
make style # Run before submitting
68+
make style # Run before submitting
6969
make fast-test # Quick test suite
7070
```
7171

72+
Optionally, `make install-pre-commit` installs git hooks so ruff and mypy run on `git commit`. Hooks do not replace `make style`: they run on staged files, while CI runs `make style` across the tree.
73+
7274
## Coding Standards
7375

7476
- Run `make style` before submitting a pull request

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,9 @@ risingwave-test: engine-risingwave-up
222222

223223
starrocks-test: engine-starrocks-up
224224
pytest -n auto -m "starrocks" --reruns 3 --junitxml=test-results/junit-starrocks.xml
225+
226+
db2-test: engine-db2-up
227+
pytest -n auto -m "db2" --reruns 3 --junitxml=test-results/junit-db2.xml
225228

226229
#################
227230
# Cloud Engines #

‎docs/concepts/tests.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -463,6 +463,37 @@ You can also run tests that match a pattern or substring using a glob pathname e
463463
$ sqlmesh test tests/test_*
464464
```
465465

466+
Passing the path of a model file runs the tests for that model, which is useful for commit hooks and other tools that work with changed files rather than test names:
467+
468+
```
469+
$ sqlmesh test models/full_model.sql
470+
```
471+
472+
Model files and test files can be mixed, and the results are unioned. A test selected by more than one argument still runs only once, so the following runs each of `full_model`'s tests a single time even though both arguments cover them:
473+
474+
```
475+
$ sqlmesh test models/full_model.sql tests/test_full_model.yaml
476+
```
477+
478+
An argument that is neither a known model file nor a known test file is an error, so a mistyped or stale path fails instead of quietly running no tests. A model that simply has no tests is not an error.
479+
480+
You can pass `--local` to run tests without loading state from the configured state connection:
481+
482+
``` bash
483+
$ sqlmesh test --local
484+
```
485+
486+
This keeps offline runs and commit hooks from opening a connection to the state backend.
487+
488+
In multi-repository setups, or when running tests for only a subset of projects, models that exist only in remote state are not loaded under `--local`. Unlike [`sqlmesh lint --local`](../guides/linter.md), which reports additional errors in that situation, a test whose model is missing is **skipped with a warning and the run still succeeds**:
489+
490+
```
491+
[WARNING] Model '"memory"."bronze"."a"' was not found at tests/test_a.yaml
492+
.**Successfully Ran `1` Tests Against `duckdb`**
493+
```
494+
495+
So a passing exit code alone does not mean every test you expected actually ran. Watch the output for these warnings, and keep in mind that a hook using `--local` will not fail on them.
496+
466497
### Testing using notebooks
467498

468499
You can execute tests on demand using the `%run_test` notebook magic as follows:

‎docs/development.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Once you have activated your virtual environment, you can install the dependenci
4242
make install-dev
4343
```
4444

45-
Optionally, you can use pre-commit to automatically run linters/formatters:
45+
Optionally, `make install-pre-commit` installs git hooks so ruff and mypy run on `git commit`. Hooks do not replace `make style`: they run on staged files, while CI runs `make style` across the tree.
4646

4747
```bash
4848
make install-pre-commit

0 commit comments

Comments
 (0)