Skip to content
#

incremental-data-load

Here are 2 public repositories matching this topic...

This project demonstrates a data pipeline using Snowflake, Google Cloud Storage (GCS), and Pub/Sub for loading data incrementally into Snowflake tables. The pipeline is triggered by new files arriving in a specified GCS bucket, notifying Snowflake through Pub/Sub, and subsequently updating tables in Snowflake.

  • Updated Jan 3, 2024

Built 7 production-grade ADF pipeline patterns including CDC incremental load (JSON & DB-based audit logging), dual REST API pagination, dynamic file routing, dynamic schema mapping, Delta Lake upserts via Mapping Data Flow, and multi-table orchestration driven by a SQL control table with parallel execution

  • Updated Sep 13, 2026

Add this topic to your repo

To associate your repository with the incremental-data-load topic, visit your repo's landing page and select "manage topics."

Learn more