Skip to content

Commit 5774894

Browse files
davidzajac1davidzajac1
authored
Makefile and CI upgrades (#120)
1 parent 2450b5e commit 5774894

5 files changed

Lines changed: 87 additions & 45 deletions

File tree

‎.github/workflows/generate-docs.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ on: [push]
55
env:
66
DBT_PROFILES_DIR: ./
77
DBT_VERSION: 1.7
8+
PYTHON_VERSION: "3.8.x"
89

910
jobs:
1011
generate-docs:
@@ -30,7 +31,7 @@ jobs:
3031

3132
- uses: actions/setup-python@v4
3233
with:
33-
python-version: "3.8.x"
34+
python-version: ${{ env.PYTHON_VERSION }}
3435

3536
- name: Install dependencies and run
3637
run: |

‎.github/workflows/run-db-tests.yml‎

Lines changed: 41 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,48 @@ env:
1919
BIGQUERY_TESTING_AUTH_PROVIDER_X509_CERT_URL: ${{ secrets.BIGQUERY_TESTING_AUTH_PROVIDER_X509_CERT_URL }}
2020
BIGQUERY_TESTING_CLIENT_X509_CERT_URL: ${{ secrets.BIGQUERY_TESTING_CLIENT_X509_CERT_URL }}
2121
DBT_VERSION: 1.7
22+
PYTHON_VERSION: "3.8.x"
2223

2324
jobs:
25+
test-postgres:
26+
runs-on: ubuntu-latest
27+
if: github.event_name == 'push'
28+
services:
29+
postgres:
30+
image: postgres
31+
env:
32+
POSTGRES_PASSWORD: postgres
33+
# Set health checks to wait until postgres has started
34+
options: >-
35+
--health-cmd pg_isready
36+
--health-interval 10s
37+
--health-timeout 5s
38+
--health-retries 5
39+
ports:
40+
# Maps tcp port 5432 on service container to the host
41+
- 5432:5432
42+
steps:
43+
- name: Check out
44+
uses: actions/checkout@v2
45+
46+
- uses: actions/setup-python@v4
47+
with:
48+
python-version: ${{ env.PYTHON_VERSION }}
49+
50+
- name: Install dependencies
51+
working-directory: ./integration_tests
52+
run: |
53+
pip install -r requirements.txt
54+
pip install dbt-postgres==$DBT_VERSION
55+
dbt deps
56+
57+
- name: Test DB
58+
working-directory: ./integration_tests/python_tests
59+
run: pytest --db postgres --source_schema dq
60+
2461
test-other-dbs:
2562
runs-on: ubuntu-latest
26-
if: github.event_name == 'push' && github.repository == 're-data/dbt-re-data'
63+
if: github.event_name == 'push' && github.repository == 're-data/dbt-re-data' && github.ref == 'refs/heads/master'
2764
strategy:
2865
fail-fast: false
2966
matrix:
@@ -34,7 +71,7 @@ jobs:
3471

3572
- uses: actions/setup-python@v4
3673
with:
37-
python-version: "3.8.x"
74+
python-version: ${{ env.PYTHON_VERSION }}
3875

3976
- name: Inject slug/short variables
4077
uses: rlespinasse/github-slug-action@v3.x
@@ -71,45 +108,9 @@ jobs:
71108
run: |
72109
pytest --db ${{ matrix.database }} --source_schema ${{ env.DQ_SCHEMA }}
73110
74-
test-postgres:
75-
runs-on: ubuntu-latest
76-
if: github.event_name == 'push'
77-
services:
78-
postgres:
79-
image: postgres
80-
env:
81-
POSTGRES_PASSWORD: postgres
82-
# Set health checks to wait until postgres has started
83-
options: >-
84-
--health-cmd pg_isready
85-
--health-interval 10s
86-
--health-timeout 5s
87-
--health-retries 5
88-
ports:
89-
# Maps tcp port 5432 on service container to the host
90-
- 5432:5432
91-
steps:
92-
- name: Check out
93-
uses: actions/checkout@v2
94-
95-
- uses: actions/setup-python@v4
96-
with:
97-
python-version: "3.8.x"
98-
99-
- name: Install dependencies
100-
working-directory: ./integration_tests
101-
run: |
102-
pip install -r requirements.txt
103-
pip install dbt-postgres==$DBT_VERSION
104-
dbt deps
105-
106-
- name: Test DB
107-
working-directory: ./integration_tests/python_tests
108-
run: pytest --db postgres --source_schema dq
109-
110111
clean-up-schemas:
111112
runs-on: ubuntu-latest
112-
if: github.event_name == 'delete' && github.repository == 're-data/dbt-re-data'
113+
if: github.event_name == 'delete' && github.repository == 're-data/dbt-re-data' && github.ref == 'refs/heads/master'
113114
strategy:
114115
fail-fast: false
115116
matrix:
@@ -120,7 +121,7 @@ jobs:
120121

121122
- uses: actions/setup-python@v4
122123
with:
123-
python-version: "3.8.x"
124+
python-version: ${{ env.PYTHON_VERSION }}
124125

125126
- name: Inject slug/short variables
126127
uses: rlespinasse/github-slug-action@v3.x

‎Makefile‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
2+
# Maybe change this if you are not running on a Mac
3+
CONTAINER_ARCH = linux/amd64
4+
5+
.PHONY: help run-all-ci test-postgres generate-docs
6+
7+
help:
8+
$(info ${HELP_MESSAGE})
9+
@exit 0
10+
11+
12+
# Run GitHub Actions CI jobs locally
13+
run-all-ci: test-postgres generate-docs
14+
@echo "All CI steps completed."
15+
16+
test-postgres:
17+
@echo "Running test-postgres job..."
18+
act -j test-postgres --container-architecture $(CONTAINER_ARCH)
19+
20+
generate-docs:
21+
@echo "Running generate-docs job..."
22+
act -j generate-docs --container-architecture $(CONTAINER_ARCH)
23+
24+
25+
define HELP_MESSAGE
26+
Usage: $ make [TARGETS]
27+
28+
TARGETS
29+
help Shows this help message
30+
run-all-ci Runs all CI steps
31+
test-postgres Runs test-postgres job
32+
generate-docs Generates documentation
33+
34+
endef

‎README.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,26 +2,30 @@
22

33
re_data is an open-source data reliability framework for the modern data stack. 😊
44

5-
Currently, re_data focuses on observing the dbt project (together with underlaying data warehouse - Postgres, BigQuery, Snowflake, Redshift).
5+
Currently, re_data focuses on observing the dbt project (together with underlying data warehouse - Postgres, BigQuery, Snowflake, Redshift).
66

77
Data transformations in re_data are implemented and exposed as models & macros in this dbt package.
88

99
# Live demo
1010

11-
Check out our **[live demo](https://getre.io/demo)** of what re_data can do for you 😊
11+
Check out our **[live demo](https://docs.getre.io/ui-latest)** of what re_data can do for you 😊
1212

1313
# Getting started
1414

1515
[Check our docs!](https://docs.getre.io/) 🙂
1616

17-
[Join re_data community on Slack](https://www.getre.io/slack) (we are very responsive there)
17+
[Join re_data community on Slack](https://join.slack.com/t/re-data/shared_invite/zt-vkauq1y8-tL4R4_H5nZoVvyXyy0hdug) (we are very responsive there)
1818

1919
[Check out more info, issues, etc. in master repo](https://github.com/re-data/re-data)
2020

2121
# Community
2222

2323
Say, hi to us on! 🙂
2424

25-
- [Slack](https://www.getre.io/slack)
25+
- [Slack](https://join.slack.com/t/re-data/shared_invite/zt-vkauq1y8-tL4R4_H5nZoVvyXyy0hdug)
2626
- [Twitter](https://twitter.com/re_data_labs)
2727
- [LinkedIn](https://www.linkedin.com/company/74608627/)
28+
29+
# Contributing
30+
31+
Any contributions are greatly appreciated! Most of our documentation and GitHub issues are managed in the primary [re-data](https://github.com/re-data/re-data) repo. See the Contributing section in [re-data](https://github.com/re-data/re-data) for details.

‎integration_tests/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,5 @@ target/
33
dbt_modules/
44
dbt_packages/
55
logs/
6+
.user.yml
7+
package-lock.yml

0 commit comments

Comments
 (0)