Submit CDC lakehouse reliability assignment

submission/simerjit/APPROACH.md

@@ -0,0 +1,66 @@
+# Approach
+
+## Understanding
+
+The assignment asks for a CDC lakehouse reliability pipeline. The solution should model a realistic transactional source, capture inserts/updates/deletes, preserve raw change history in a lake layer, publish current-state warehouse tables, stop on unsafe schema changes, support restore/replay, expose catalog metadata, and include validations or tests.
+
+## Proposed Solution
+
+I implemented the assignment with Databricks, PySpark, and Delta Lake using a wallet and payments domain.
+
+The source model includes:
+
+- `customers`
+- `wallet_accounts`
+- `merchants`
+- `payment_transactions`
+- `payment_attempts`
+
+CDC events are simulated as JSON files in a Databricks Volume. Each event contains:
+
+- `source_table`
+- `cdc_operation`
+- `cdc_sequence`
+- `source_commit_ts`
+- `schema_version`
+- `payload_json`
+
+## Lakehouse Design
+
+The Bronze lake table is:
+
+- `robustrade_dev.bronze.cdc_events`
+
+It is append-only and keeps every CDC event.
+
+The Silver warehouse tables are:
+
+- `robustrade_dev.silver.customers_current`
+- `robustrade_dev.silver.wallet_accounts_current`
+- `robustrade_dev.silver.merchants_current`
+- `robustrade_dev.silver.payment_transactions_current`
+- `robustrade_dev.silver.payment_attempts_current`
+
+Silver tables are maintained with Delta `MERGE` and keep the latest non-deleted row per primary key.
+
+## Reliability Plan
+
+The implementation includes:
+
+- CDC inserts, updates, and deletes.
+- Auto Loader ingestion into Bronze Delta.
+- Delta `MERGE` for Silver current-state tables.
+- Schema contract checks before Silver writes.
+- CDC correctness checks for update/delete/idempotency behavior.
+- Validation parity checks for business rules.
+- Catalog metadata.
+- Delta time travel restore demo.
+
+## Recovery Strategy
+
+Bronze Delta is append-only and replayable. Silver tables can be rebuilt from Bronze, and accidental Silver changes can be recovered with Delta Lake history using `DESCRIBE HISTORY` and `RESTORE TABLE`.
+
+## Responsible AI Usage
+
+AI assistance was used to help structure the notebooks, debug Databricks errors, and refine explanations. I personally ran the notebooks in Databricks, reviewed the outputs, corrected errors, and validated the final pipeline behavior.
+

submission/simerjit/PR_DESCRIPTION.md

@@ -0,0 +1,74 @@
+# PR: CDC Lakehouse Reliability Assignment
+
+## Summary
+
+This PR implements a CDC lakehouse reliability pipeline using Databricks, PySpark, and Delta Lake. It simulates a wallet/payments source system, writes CDC events to a Databricks Volume, ingests them into an append-only Bronze Delta table, applies schema-safe Delta merges into Silver current-state tables, runs validation checks, documents catalog metadata, and demonstrates Delta time travel restore.
+
+## Source Schema Design
+
+The source model has five tables:
+
+- `customers`
+- `wallet_accounts`
+- `merchants`
+- `payment_transactions`
+- `payment_attempts`
+
+`customers`, `wallet_accounts`, `merchants`, and `payment_transactions` are strong entities. `payment_attempts` is a weak entity because each attempt depends on a parent payment transaction.
+
+The source design includes primary keys, foreign keys, indexes, timestamps, nullable optional fields, currency fields, status domains, non-negative wallet balance checks, and positive payment amount checks.
+
+## CDC Strategy
+
+CDC is simulated as JSON files in a Databricks Volume. Each event includes `source_table`, `cdc_operation`, `cdc_sequence`, `source_commit_ts`, `schema_version`, and `payload_json`.
+
+The sample stream contains inserts, an update, and a delete. In production, this simulated stream would be replaced by database logs, Debezium, Kafka, Fivetran, or another CDC connector.
+
+## Lake And Warehouse Modeling
+
+The lake layer is `robustrade_dev.bronze.cdc_events`. It is append-only and retains full CDC history.
+
+The warehouse layer contains Silver current-state tables. These tables are built with Delta `MERGE`, ordered by `source_commit_ts` and `cdc_sequence`, and retain only the latest non-deleted row per key.
+
+## Schema Change Safety
+
+Before writing to Silver, each CDC payload is parsed using an explicit schema and checked against a contract. Missing required fields, data type changes, or required fields becoming nullable stop the pipeline before downstream mutation.
+
+## Validation Parity
+
+Warehouse checks mirror source business rules:
+
+- primary key uniqueness
+- referential integrity
+- non-negative wallet balances
+- positive payment amounts
+- accepted transaction statuses
+- duplicate CDC sequence detection
+
+## Catalog Exposure
+
+Catalog metadata is provided in `catalog.yml` and the `08_catalog_metadata` notebook. It documents dataset name, layer, platform, owner, description, consumers, primary key, and update cadence.
+
+## Historical Recovery
+
+Silver tables can be rebuilt from Bronze. The restore notebook also demonstrates Delta Lake time travel with `DESCRIBE HISTORY` and `RESTORE TABLE`.
+
+## Responsible AI Usage
+
+AI assistance was used to help structure the notebooks, debug Databricks errors, and refine explanations. I personally ran the notebooks in Databricks, reviewed the outputs, corrected errors, and validated the final pipeline behavior.
+
+## Testing And Validation
+
+Completed validations:
+
+- 7 CDC events generated.
+- 7 events loaded into Bronze Delta.
+- Schema safe-stop test failed correctly for an unsafe type change.
+- Silver merges created current-state tables.
+- `payment_transactions_current` showed transaction `t_001` as `SETTLED`.
+- CDC correctness checks passed.
+- Rerunning the Silver merge did not create duplicate rows.
+- Quality checks passed.
+- Catalog metadata coverage passed.
+- Delta restore returned a corrupted transaction status back to `SETTLED`.
+

submission/simerjit/README.md

@@ -0,0 +1,58 @@
+# CDC Lakehouse Reliability Assignment
+
+Databricks, PySpark, and Delta Lake solution for the Robustrade data engineering assignment.
+
+## Stack
+
+- Databricks
+- PySpark
+- Delta Lake
+- Unity Catalog
+- Databricks Volumes
+- YAML metadata/configuration
+
+## Run Order
+
+Run the notebooks in this order:
+
+1. `00_setup_and_source_model`
+2. `01_generate_cdc_events`
+3. `02_bronze_ingest_delta`
+4. `03_apply_silver_merges`
+5. `06_cdc_correctness_tests`
+6. `05_quality_checks`
+7. `08_catalog_metadata`
+8. `04_restore_with_delta_time_travel`
+9. `09_final_summary_or_pr_notes`
+
+## What The Project Demonstrates
+
+- Source schema design for a wallet/payments domain.
+- Simulated CDC event contract.
+- Insert, update, and delete CDC events.
+- Bronze Delta append-only history.
+- Silver Delta current-state tables.
+- Schema-safe stopping before incompatible writes.
+- CDC correctness and idempotency checks.
+- Validation parity for warehouse tables.
+- Catalog metadata.
+- Delta time travel restore.
+
+## Important Tables
+
+- `robustrade_dev.bronze.cdc_events`
+- `robustrade_dev.silver.customers_current`
+- `robustrade_dev.silver.wallet_accounts_current`
+- `robustrade_dev.silver.merchants_current`
+- `robustrade_dev.silver.payment_transactions_current`
+- `robustrade_dev.silver.payment_attempts_current`
+
+## Supporting Files
+
+- `APPROACH.md`: initial understanding and implementation approach.
+- `PR_DESCRIPTION.md`: PR-style summary mapped to assignment expectations.
+- `catalog.yml`: dataset catalog metadata.
+- `schema_contracts.yml`: expected source payload contracts.
+- `job_config.yml`: notebook task order and dependencies.
+- `sql/source_schema.sql`: source model DDL.
+

submission/simerjit/catalog.yml

@@ -0,0 +1,68 @@
+version: 1
+owner: data-platform
+domain: wallet-payments
+
+datasets:
+  - name: robustrade_dev.bronze.cdc_events
+    layer: bronze
+    platform: Databricks Delta Lake
+    description: Append-only CDC event log retaining all source inserts, updates, and deletes.
+    primary_key: source_table, cdc_sequence
+    update_cadence: per pipeline run
+    consumers:
+      - data engineering
+      - incident response
+      - backfill jobs
+
+  - name: robustrade_dev.silver.customers_current
+    layer: silver
+    platform: Databricks Delta Lake
+    description: Latest non-deleted customer state.
+    primary_key: customer_id
+    update_cadence: per pipeline run
+    consumers:
+      - analytics
+      - risk
+      - customer operations
+
+  - name: robustrade_dev.silver.wallet_accounts_current
+    layer: silver
+    platform: Databricks Delta Lake
+    description: Latest wallet balance and status.
+    primary_key: wallet_id
+    update_cadence: per pipeline run
+    consumers:
+      - analytics
+      - finance operations
+
+  - name: robustrade_dev.silver.merchants_current
+    layer: silver
+    platform: Databricks Delta Lake
+    description: Latest merchant reference state.
+    primary_key: merchant_id
+    update_cadence: per pipeline run
+    consumers:
+      - analytics
+      - risk
+
+  - name: robustrade_dev.silver.payment_transactions_current
+    layer: silver
+    platform: Databricks Delta Lake
+    description: Latest payment transaction state.
+    primary_key: transaction_id
+    update_cadence: per pipeline run
+    consumers:
+      - analytics
+      - finance
+      - reconciliation
+
+  - name: robustrade_dev.silver.payment_attempts_current
+    layer: silver
+    platform: Databricks Delta Lake
+    description: Latest payment processor attempt state.
+    primary_key: attempt_id
+    update_cadence: per pipeline run
+    consumers:
+      - payments operations
+      - incident response
+

submission/simerjit/job_config.yml

@@ -0,0 +1,53 @@
+job_name: cdc_lakehouse_reliability
+catalog: robustrade_dev
+volume_path: /Volumes/robustrade_dev/audit/landing
+
+tasks:
+  - task_key: setup
+    notebook: 00_setup_and_source_model
+    description: Create catalog and schemas, document source model.
+
+  - task_key: generate_cdc_events
+    notebook: 01_generate_cdc_events
+    description: Generate simulated CDC JSON events.
+
+  - task_key: bronze_ingest
+    notebook: 02_bronze_ingest_delta
+    depends_on:
+      - generate_cdc_events
+    description: Ingest CDC JSON into Bronze Delta table.
+
+  - task_key: silver_merge
+    notebook: 03_apply_silver_merges
+    depends_on:
+      - bronze_ingest
+    description: Validate schemas and merge CDC into Silver current-state tables.
+
+  - task_key: cdc_correctness_tests
+    notebook: 06_cdc_correctness_tests
+    depends_on:
+      - silver_merge
+    description: Verify CDC update, delete, idempotency, and duplicate handling.
+
+  - task_key: quality_checks
+    notebook: 05_quality_checks
+    depends_on:
+      - silver_merge
+    description: Run warehouse validation parity checks.
+
+  - task_key: catalog_metadata
+    notebook: 08_catalog_metadata
+    depends_on:
+      - silver_merge
+    description: Document lake and warehouse datasets.
+
+  - task_key: restore_demo
+    notebook: 04_restore_with_delta_time_travel
+    depends_on:
+      - silver_merge
+    description: Demonstrate Delta history and restore.
+
+  - task_key: final_summary
+    notebook: 09_final_summary_or_pr_notes
+    description: Summarize assignment solution and validation results.
+

submission/simerjit/notebooks/00_setup_and_source_model.py

@@ -0,0 +1,66 @@
+# Databricks notebook source
+# MAGIC %sql
+# MAGIC CREATE CATALOG IF NOT EXISTS robustrade_dev;
+# MAGIC USE CATALOG robustrade_dev;
+# MAGIC
+# MAGIC CREATE SCHEMA IF NOT EXISTS bronze;
+# MAGIC CREATE SCHEMA IF NOT EXISTS silver;
+# MAGIC CREATE SCHEMA IF NOT EXISTS audit;
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC Catalog Check   
+
+# COMMAND ----------
+
+# MAGIC %sql
+# MAGIC SHOW CATALOGS;
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC Schema Check
+
+# COMMAND ----------
+
+# MAGIC %sql
+# MAGIC SHOW SCHEMAS IN robustrade_dev;
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC # Source Model
+# MAGIC
+# MAGIC This project models a wallet and payments source system with five tables:
+# MAGIC
+# MAGIC - customers
+# MAGIC - wallet_accounts
+# MAGIC - merchants
+# MAGIC - payment_transactions
+# MAGIC - payment_attempts
+# MAGIC
+# MAGIC Strong entities:
+# MAGIC - customers
+# MAGIC - wallet_accounts
+# MAGIC - merchants
+# MAGIC - payment_transactions
+# MAGIC
+# MAGIC Weak entity:
+# MAGIC - payment_attempts
+# MAGIC
+# MAGIC payment_attempts is weak because each attempt depends on a parent payment transaction.
+# MAGIC
+# MAGIC Relationships:
+# MAGIC - customers -> wallet_accounts
+# MAGIC - customers -> payment_transactions
+# MAGIC - wallet_accounts -> payment_transactions
+# MAGIC - merchants -> payment_transactions
+# MAGIC - payment_transactions -> payment_attempts
+# MAGIC
+# MAGIC Important invariants:
+# MAGIC - wallet balances must be non-negative
+# MAGIC - payment amounts must be positive
+# MAGIC - statuses must be from allowed values
+# MAGIC - payments must reference valid customers, wallets, and merchants
+# MAGIC - attempts must reference valid payment transactions