-
Notifications
You must be signed in to change notification settings - Fork 1
Clean and fine-tune the examples repo: problem-first READMEs (TOW-2417) #6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
giray123
wants to merge
6
commits into
main
Choose a base branch
from
feature/tow-2417-tower-examples-fine-tuning-single-repo-cli-templates
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
30468f1
Add a problem-first root README indexing all examples
giray123 4b14a65
Fill in the empty READMEs for 09 and 16; fix README casing in 15
giray123 d8422cf
Lead the ticker examples' READMEs with the problem they solve
giray123 7d52a84
Remove the WIP data-pond stub and unreferenced root data files
giray123 d7e5609
Clarify that the Iceberg catalog is Tower-hosted with one-click setup
giray123 de87329
Address CodeRabbit review: accurate LLM data-flow claim, secrets hygi…
giray123 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| # Query Iceberg Tables with DuckDB | ||
|
|
||
| You have data in an Iceberg lakehouse and just want to run SQL on it — without standing up a query engine. This app connects DuckDB to a Tower Iceberg catalog and runs a query against the `daily_ticker_data` table (populated by [05-write-ticker-data-to-iceberg](../05-write-ticker-data-to-iceberg)). | ||
|
|
||
| ## What It Does | ||
|
|
||
| 1. Loads DuckDB's `iceberg` and `httpfs` extensions | ||
| 2. Creates a DuckDB `SECRET` from your Iceberg REST catalog credentials | ||
| 3. Attaches the catalog as a database and runs a plain `SELECT` against it | ||
|
|
||
| Swap the query at the bottom of [main.py](./main.py) for your own SQL. | ||
|
|
||
| ## What You Need | ||
|
|
||
| - A Tower-hosted Iceberg catalog containing data (run example 05 first — nothing external to set up) | ||
| - Three Tower secrets with your Tower catalog's REST credentials: | ||
|
|
||
| ```bash | ||
| tower secrets create --name=IRC_CLIENT_ID --value="[YOUR_CLIENT_ID]" | ||
| tower secrets create --name=IRC_CLIENT_SECRET --value="[YOUR_CLIENT_SECRET]" | ||
| tower secrets create --name=IRC_ENDPOINT --value="[YOUR_CATALOG_ENDPOINT]" | ||
| ``` | ||
|
|
||
| ## Run It | ||
|
|
||
| ```bash | ||
| cd 09-run-duckdb-queries-on-iceberg | ||
| tower deploy | ||
| tower run | ||
| ``` | ||
|
|
||
| Add `--local` to `tower run` to execute on your machine while still reading secrets from Tower. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # Replicate Data into Snowflake with Sling | ||
|
|
||
| You need to move data into a warehouse without writing pipeline code. This app uses [Sling](https://slingdata.io/) to replicate a local JSON file into a Snowflake table from a declarative YAML config — the Python is just a thin runner. | ||
|
|
||
| ## What It Does | ||
|
|
||
| 1. Reads the replication config from [sling.yaml](./sling.yaml) (source: the bundled [data/input.json](./data/input.json); target: `PUBLIC.SLING_JSON_TOWER_DEMO`) | ||
| 2. Runs the replication in `full-refresh` mode | ||
|
|
||
| Point `sling.yaml` at your own sources and targets — Sling supports databases, object stores, and files. | ||
|
|
||
| ## What You Need | ||
|
|
||
| A Tower secret named `SNOWFLAKE_URL` with your Snowflake connection string, in [Sling's URL format](https://docs.slingdata.io/connections/database-connections/snowflake): | ||
|
|
||
| ```bash | ||
| tower secrets create --name=SNOWFLAKE_URL --value="snowflake://user:password@account/database?schema=PUBLIC&warehouse=COMPUTE_WH" | ||
| ``` | ||
|
|
||
| > **Note:** Inline values like this end up in your shell history. Use a throwaway or sandbox Snowflake user for testing, and avoid pasting production credentials into commands. | ||
|
|
||
| ## Run It | ||
|
|
||
| ```bash | ||
| cd 16-sling-data | ||
| tower deploy | ||
| tower run | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| # Tower Examples | ||
|
|
||
| Working example apps for [Tower](https://tower.dev), each built around a common data problem. Every example is a complete Tower app — a few Python files plus a `Towerfile` — that you can clone, deploy, and run in minutes, then adapt to your own stack. | ||
|
|
||
| ## Quick start | ||
|
|
||
| Install the [Tower CLI](https://docs.tower.dev/getting-started/installation), sign in, and run the first example — it needs no secrets and no setup: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/tower/tower-examples.git | ||
| cd tower-examples/01-hello-world | ||
| tower login | ||
| tower deploy | ||
| tower run | ||
| ``` | ||
|
|
||
| Then pick the example below that matches the problem you're actually trying to solve. | ||
|
|
||
| ## Find your problem | ||
|
|
||
| ### Start here — runs with zero setup | ||
|
|
||
| | Example | The problem it solves | What you need | | ||
| |---|---|---| | ||
| | [01-hello-world](./01-hello-world) | See the whole Tower workflow — deploy, run, parameters, logs — in two minutes | Nothing | | ||
| | [15-interactive-marimo-notebook](./15-interactive-marimo-notebook) | Serve an interactive Python notebook as a running app, no infra to host | Nothing | | ||
|
|
||
| ### Load files into a warehouse | ||
|
|
||
| | Example | The problem it solves | What you need | | ||
| |---|---|---| | ||
| | [02-dlthub-s3-to-snowflake](./02-dlthub-s3-to-snowflake) | CSVs land in S3 and need to end up in Snowflake, incrementally and reliably (dlt) | Snowflake credentials — the source bucket is public | | ||
| | [03-dlthub-s3-to-motherduck](./03-dlthub-s3-to-motherduck) | Same S3 ingestion problem, targeting MotherDuck (dlt) | MotherDuck token — the source bucket is public | | ||
| | [16-sling-data](./16-sling-data) | Replicate local/JSON data into Snowflake with a declarative YAML config (Sling) | Snowflake connection secret | | ||
|
|
||
| ### Build a lakehouse on Apache Iceberg | ||
|
|
||
| These examples form a small end-to-end lakehouse: ingest → analyze → query → maintain. They share one prerequisite: an Iceberg catalog named `default` — but Tower hosts this catalog for you, so there is nothing external to sign up for. Deploying an example from the [Tower app](https://app.tower.dev) creates it in one click (the same screen can fill required secrets with Tower sandbox values for testing); on the CLI path, create it once in the Tower UI. | ||
|
|
||
| | Example | The problem it solves | What you need | | ||
| |---|---|---| | ||
| | [05-write-ticker-data-to-iceberg](./05-write-ticker-data-to-iceberg) | Pull data from an external API on a schedule and land it in an Iceberg table (demo data: stock prices) | Tower-hosted catalog — no API keys | | ||
| | [06-analyze-ticker-data-in-iceberg](./06-analyze-ticker-data-in-iceberg) | Run LLM-assisted analysis over data already in your lakehouse | Tower-hosted catalog + inference key (or Tower's sandbox key in-app) | | ||
| | [09-run-duckdb-queries-on-iceberg](./09-run-duckdb-queries-on-iceberg) | Query Iceberg tables with plain SQL from DuckDB | Tower-hosted catalog + its REST credentials as secrets | | ||
| | [11-trim-ticker-table](./11-trim-ticker-table) | Enforce a retention window by deleting old rows from an Iceberg table | Tower-hosted catalog with data (run 05 first) | | ||
| | [17-list-catalog-tables](./17-list-catalog-tables) | Inspect what namespaces and tables exist in a catalog | Tower-hosted catalog | | ||
| | [18-read-table-rows](./18-read-table-rows) | Peek at the first rows of any Iceberg table | Tower-hosted catalog with data | | ||
|
|
||
| ### Orchestrate many runs | ||
|
|
||
| | Example | The problem it solves | What you need | | ||
| |---|---|---| | ||
| | [08-fan-out-ticker-runs](./08-fan-out-ticker-runs) | Fan one job out into parallel runs and wait for them all (Tower `run`/`wait`) | Example 05 deployed | | ||
|
|
||
| ### Put LLMs and agents to work on your data | ||
|
|
||
| | Example | The problem it solves | What you need | | ||
| |---|---|---| | ||
| | [07-deepseek-summarize-github](./07-deepseek-summarize-github) | Feed operational data (GitHub issues) to an LLM and get an actionable recommendation back | Tower-hosted catalog + inference API key | | ||
| | [13-ticker-update-agent](./13-ticker-update-agent) | Deploy an AI agent that answers questions from — and maintains — business data in your lakehouse | Tower-hosted catalog + `OPENAI_API_KEY` (or Tower's sandbox key in-app) | | ||
|
|
||
| ### Run dbt in production | ||
|
|
||
| | Example | The problem it solves | What you need | | ||
| |---|---|---| | ||
| | [14-dbt-core-ecommerce-analytics](./14-dbt-core-ecommerce-analytics) | Run a real dbt Core project (seed → build) as a deployable, schedulable app | `DBT_PROFILE_YAML` secret with your warehouse profile | | ||
|
|
||
| ## How every example works | ||
|
|
||
| Each directory is a self-contained Tower app: | ||
|
|
||
| - **`Towerfile`** — declares the app: name, entrypoint script, source files, and runtime parameters. | ||
| - **A Python script** — ordinary Python; no framework to learn. | ||
| - **`pyproject.toml`** — dependencies, installed automatically at run time. | ||
|
|
||
| The workflow is always the same: `tower deploy` from the example's directory, then `tower run` (add `--local` to execute on your machine while still using Tower secrets and catalogs). Set secrets with `tower secrets create`; each example's README lists exactly which ones it needs. | ||
|
|
||
| ## Learn more | ||
|
|
||
| - [Tower documentation](https://docs.tower.dev) — concepts, CLI reference, guides | ||
| - [Tower Control](https://control.tower.dev) — describe an app in natural language and let the agent build and deploy it | ||
| - [tower.dev](https://tower.dev) — what Tower is and who it's for |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.