Repository navigation
add localstack-bruin: Bruin data pipeline on LocalStack S3 + Snowflake - #15
Merged
Merged
Conversation
Sample Bruin pipeline that loads raw CSVs from LocalStack S3 into the LocalStack Snowflake emulator (external stage + COPY INTO), transforms them with data quality checks, and writes a report back to S3. Includes a CI workflow that runs the pipeline end to end and checks the report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssing license Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run LocalStack (S3) and localstack/snowflake-next side by side, with external stages reaching S3 via SF_S3_ENDPOINT. This drops the earlier emulator workarounds: COPY uses a stage-level file format with MATCH_BY_COLUMN_NAME, and the report is unloaded from Snowflake to S3 with COPY INTO @stage instead of boto3. Ports are configurable via LOCALSTACK_PORT / SNOWFLAKE_PORT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New sample in
localstack-bruin/that runs a Bruin pipeline fully locally against LocalStack, using both AWS (S3) and the Snowflake emulator (localstack/snowflake-next).raw.shop_data: Snowflake external stage on a LocalStack S3 bucket,COPY INTO ... MATCH_BY_COLUMN_NAMEfrom CSVanalytics.orders_enriched: join in Snowflake plus data quality checks that fail the run on bad dataanalytics.revenue_by_country: aggregation in Snowflake, unloaded as JSON straight to LocalStack S3make all/make testfor local runs, plus.github/workflows/test-localstack-bruin.ymlrunning the flow end to end in CINotes
localstack/localstack-pro(S3, portLOCALSTACK_PORT, default 4566) andlocalstack/snowflake-next(portSNOWFLAKE_PORT, default 4567). Snowflake reaches S3 for external stages viaSF_S3_ENDPOINT=http://localstack:4566.hostoption (always*.snowflakecomputing.com), so the Snowflake steps are Bruin Python assets that get the connection injected as a secret and connect tosnowflake.localhost.localstack.cloud..bruin.ymlis committed on purpose since it only holds LocalStack's dummytestcredentials. Bruin auto-adds it to.gitignore, so the sample's.gitignorere-includes it..gitignoregets Bruin'slogs/entries (Bruin appends them on first run).TEST_LOCALSTACK_SNOWFLAKE_AUTH_TOKENsecret, since the default test token has no Snowflake license.make waitfails fast if the license message shows up.Test plan
snowflake-next, reruns idempotent, report totals match the source CSVs🤖 Generated with Claude Code