You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This project demonstrates a data pipeline using Snowflake, Google Cloud Storage (GCS), and Pub/Sub for loading data incrementally into Snowflake tables. The pipeline is triggered by new files arriving in a specified GCS bucket, notifying Snowflake through Pub/Sub, and subsequently updating tables in Snowflake.
Built 7 production-grade ADF pipeline patterns including CDC incremental load (JSON & DB-based audit logging), dual REST API pagination, dynamic file routing, dynamic schema mapping, Delta Lake upserts via Mapping Data Flow, and multi-table orchestration driven by a SQL control table with parallel execution