Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ PGPASSWORD=
# ----- Estuary ---------------------------------------------------------------
# Refresh/CLI token from the Estuary dashboard: Admin -> CLI-API -> Generate token.
ESTUARY_TOKEN=
# Your tenant prefix plus a demo namespace. All Flow tasks live under this.
# Your tenant prefix plus a demo namespace. All Estuary tasks live under this.
ESTUARY_PREFIX= # e.g. yourTenant/estuary-cdc-demo

# ----- Snowflake -------------------------------------------------------------
Expand Down
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ shadowtraffic/license.env
# Claude Code working config (session permission allowlists, not project config)
.claude/

# Personal pulled Flow catalog (contains SOPS-encrypted creds + real infra IDs).
# Personal pulled Estuary catalog (contains SOPS-encrypted creds + real infra IDs).
# The shareable catalog template lives in flowctl/flow.yaml.
/flow.yaml
wassDemo/

# Rendered Flow catalog (contains substituted credentials)
# Rendered Estuary catalog (contains substituted credentials)
flowctl/flow.generated.yaml

# Terraform local state & working dir
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Estuary CDC Demo — Postgres → Estuary Flow → Snowflake
# Estuary CDC Demo — Postgres → Estuary → Snowflake

A self-contained, one-command demo of real-time change data capture. Terraform
stands up an RDS Postgres instance, ShadowTraffic streams realistic inserts and
updates into it, Estuary Flow captures the changes via logical replication, and
updates into it, Estuary captures the changes via logical replication, and
a Snowflake materialization lands them in append-only tables.

Everything spins up with `./scripts/start.sh` and tears down cleanly with
Expand Down Expand Up @@ -31,7 +31,7 @@ Everything spins up with `./scripts/start.sh` and tears down cleanly with
│ slot: flow_slot · pub: flow_publication
┌────────────────────────────────────┐
│ Estuary Flow
│ Estuary
│ source-postgres (capture) │
│ │ │
│ ▼ │
Expand Down Expand Up @@ -92,7 +92,7 @@ Accounts:

- **AWS** — any account; the instance is free-tier eligible (`db.t3.micro`, 20 GB).
- **ShadowTraffic** — free license from https://shadowtraffic.io/pricing.html
- **Estuary Flow** — https://dashboard.estuary.dev (note your tenant prefix).
- **Estuary** — https://dashboard.estuary.dev (note your tenant prefix).
- **Snowflake** — a trial account works; this demo uses an X-SMALL warehouse.

## First-time setup
Expand Down Expand Up @@ -221,9 +221,9 @@ CREATE WAREHOUSE IF NOT EXISTS <SNOWFLAKE_WAREHOUSE>
```

This runs Terraform → reads outputs → sets up Postgres CDC → starts
ShadowTraffic → publishes the Flow catalog. Re-running it is safe (idempotent).
ShadowTraffic → publishes the Estuary catalog. Re-running it is safe (idempotent).

If the Flow capture, collections, and materialization already exist under your
If the capture, collections, and materialization already exist under your Estuary
prefix, `start.sh` detects them (via `flowctl catalog list`) and **skips the
publish** so a running pipeline isn't disturbed — `flowctl` would update them in
place anyway, never rename or recreate them. To force a republish (e.g. after
Expand Down Expand Up @@ -286,7 +286,7 @@ frequently (more credits); to batch for cost instead, raise `syncFrequency` (e.g
./scripts/teardown.sh
```

In order: stop ShadowTraffic → delete the Flow capture/collections/materialization
In order: stop ShadowTraffic → delete the Estuary capture/collections/materialization
→ `DROP SCHEMA IF EXISTS <DB>.<SCHEMA> CASCADE` in Snowflake (via `snowsql`) →
`terraform destroy -auto-approve`. Steps 1–3 are best-effort so teardown always
reaches the RDS destroy.
Expand Down
2 changes: 1 addition & 1 deletion flowctl/flow.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Estuary Flow catalog: Postgres CDC capture -> 3 collections -> Snowflake.
# Estuary catalog: Postgres CDC capture -> 3 collections -> Snowflake.
#
# This is a TEMPLATE. scripts/start.sh renders the ${...} placeholders with
# `envsubst` into flowctl/flow.generated.yaml (git-ignored), then publishes
Expand Down
10 changes: 5 additions & 5 deletions scripts/start.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
# Full startup orchestration for the Postgres -> Estuary Flow -> Snowflake
# Full startup orchestration for the Postgres -> Estuary -> Snowflake
# CDC demo. Idempotent and safe to re-run.
#
# 1. terraform apply -> provision RDS Postgres
Expand Down Expand Up @@ -153,9 +153,9 @@ info "ShadowTraffic running. Tail logs with:"
info " docker compose -f shadowtraffic/docker-compose.yml logs -f"

# ---------------------------------------------------------------------------
# 5. Deploy the Estuary Flow catalog
# 5. Deploy the Estuary catalog
# ---------------------------------------------------------------------------
step "5/5 Deploying Estuary Flow catalog"
step "5/5 Deploying Estuary catalog"

# Pin flowctl to THIS token's identity for every command below.
#
Expand Down Expand Up @@ -210,7 +210,7 @@ for _spec in "${EXPECTED_SPECS[@]}"; do
done

if [ "${#missing_specs[@]}" -eq 0 ] && [ "${FORCE_PUBLISH:-0}" != "1" ]; then
info "Flow catalog already deployed under ${ESTUARY_PREFIX}/ — all 5 specs present."
info "Estuary catalog already deployed under ${ESTUARY_PREFIX}/ — all 5 specs present."
info "Skipping publish so the running pipeline isn't disturbed (FORCE_PUBLISH=1 to republish)."
else
if [ "${#missing_specs[@]}" -gt 0 ]; then
Expand Down Expand Up @@ -248,7 +248,7 @@ step "Done."
cat <<EOF

The pipeline is live:
Postgres (RDS) --CDC--> Estuary Flow --delta-updates--> Snowflake
Postgres (RDS) --CDC--> Estuary --delta-updates--> Snowflake

Collection storage (OPTIONAL later lab exercise — the demo already works on
Estuary's default managed storage, so nothing here is required now):
Expand Down
8 changes: 4 additions & 4 deletions scripts/teardown.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# order of creation, so nothing is left running (and billing).
#
# 1. docker compose down -> stop ShadowTraffic
# 2. flowctl delete -> remove Flow capture/collections/materialization
# 2. flowctl delete -> remove Estuary capture/collections/materialization
# 3. DROP SCHEMA ... CASCADE -> remove Snowflake schema + tables
# 4. terraform destroy -> tear down RDS Postgres
#
Expand Down Expand Up @@ -54,9 +54,9 @@ else
fi

# ---------------------------------------------------------------------------
# 2. Delete the Estuary Flow catalog
# 2. Delete the Estuary catalog
# ---------------------------------------------------------------------------
step "2/4 Deleting Estuary Flow resources"
step "2/4 Deleting Estuary resources"
if [ -n "${ESTUARY_TOKEN:-}" ] && [ -n "${ESTUARY_PREFIX:-}" ] && command -v flowctl >/dev/null 2>&1; then
# Pin to ESTUARY_TOKEN's identity, isolated from your default flowctl login:
# authenticate a dedicated, per-tenant profile with the .env token and pass
Expand Down Expand Up @@ -92,7 +92,7 @@ if [ -n "${ESTUARY_TOKEN:-}" ] && [ -n "${ESTUARY_PREFIX:-}" ] && command -v flo
warn "administers ('flowctl auth roles list') and re-run, or delete in the UI."
fi
else
warn "ESTUARY_TOKEN/ESTUARY_PREFIX/flowctl missing; skipping Flow cleanup."
warn "ESTUARY_TOKEN/ESTUARY_PREFIX/flowctl missing; skipping Estuary cleanup."
fi

# ---------------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion terraform/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ variable "allowed_cidr" {

variable "estuary_cidrs" {
description = <<-EOT
Estuary Flow data-plane egress IPs that must reach Postgres for the CDC
Estuary data-plane egress IPs that must reach Postgres for the CDC
capture to connect. These are separate from your own IP. The default covers
all of Estuary's public data planes; trim to just your tenant's data plane
(shown under Admin -> "Allowlist IP addresses" in the dashboard;
Expand Down