Added the data engineering solution - for ecommerce domain

submission/my_submission/.gitignore

@@ -0,0 +1,153 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+Pipfile.lock
+
+# PEP 582
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# IDE settings
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+Thumbs.db
+
+# DuckDB
+*.duckdb
+*.duckdb-shm
+*.duckdb-wal
+
+# Local test databases
+test.db
+test_*.db
+
+# OS
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+
+# Project specific
+.env.local
+*.pid

submission/my_submission/APPROACH.md

@@ -0,0 +1,92 @@
+# CDC Lakehouse Assignment - E-Commerce Order Management System
+
+## Domain Choice
+
+The source system models e-commerce order processing because it gives a realistic mix of independent entities, lifecycle-dependent entities, monetary values, timestamps, status domains, and cross-table business rules.
+
+## Source Schema
+
+Strong entities:
+- `customers`
+- `products`
+- `orders`
+
+Weak entities:
+- `order_items`
+- `inventory_logs`
+- `shipments`
+- `returns`
+
+Key invariants:
+- `order.total_amount = SUM(order_items.line_total)`
+- `product.stock_qty >= 0`
+- `order_item.line_total = quantity * unit_price`
+- `shipment.delivered_at >= shipment.shipped_at`
+- `return.refund_amount <= order_item.line_total`
+- Status fields must stay inside documented enum domains.
+
+## CDC Strategy
+
+`CDCCapture` simulates WAL-style CDC. Every insert, update, and delete produces a `CDCRecord` with:
+- `sequence`
+- `operation`
+- `table`
+- `primary_key`
+- `data`
+- `captured_at`
+
+In production this would map to Debezium/Postgres WAL or a similar database log reader. The `sequence` acts like a local LSN/Kafka offset and supports checkpoint-based replay.
+
+## Lake Model
+
+`lake_cdc_events` is the immutable event log. It stores every CDC event with sequence, operation, table name, primary key, JSON row data, and capture timestamp.
+
+The lake is the historical source of truth. Retries are made idempotent by skipping events whose sequence already exists in the lake.
+
+## Warehouse Model
+
+The warehouse contains current-state tables only:
+- `wh_customers`
+- `wh_products`
+- `wh_orders`
+- `wh_order_items`
+- `wh_inventory_logs`
+- `wh_shipments`
+- `wh_returns`
+
+Each row includes:
+- `_cdc_seq`: latest CDC sequence applied to the row
+- `_deleted`: soft-delete flag for source deletes
+
+Historical recovery does not use SCD2 tables. Instead, the warehouse can be restored by clearing current-state tables and replaying `lake_cdc_events` up to a target sequence. Point-in-time queries replay lake events up to a target timestamp and fold them into row state.
+
+## Schema Change Safety
+
+`SCHEMA_CONTRACT` documents the expected source columns. `scripts/check_schema_contracts.py` validates the source schema before ingestion and fails closed if required columns are missing. This prevents silent ingestion under broken downstream assumptions.
+
+Production hardening would extend this contract to compare data types, nullability, check domains, and key constraints.
+
+## Validation Parity
+
+Source validations are represented through DuckDB constraints where practical and mirrored downstream with `scripts/run_data_quality_checks.py`.
+
+The quality checks cover:
+- Primary-key uniqueness
+- Referential integrity
+- Required fields
+- Enum domains
+- Positive monetary amounts
+- Line-item totals
+- Order totals
+- Shipment date ordering
+- Return refund limits
+
+## Catalog Exposure
+
+`catalog/catalog.json` publishes the lake dataset and all warehouse datasets with layer, owner, consumers, update cadence, schema metadata, descriptions, lineage, and validation rules.
+
+## Tradeoffs
+
+This solution uses local DuckDB and simulated CDC to keep the assignment runnable in a small environment. Production equivalents would use a relational source, WAL-based CDC, Kafka or a managed stream, object storage with table formats such as Delta/Iceberg/Hudi, and a warehouse merge job.
+
+The restore model deliberately uses lake replay instead of SCD2 to keep one authoritative history path and avoid drift between separate history tables and the raw CDC log.

submission/my_submission/IMPLEMENTATION_PLAN.md

@@ -0,0 +1,56 @@
+# CDC Lakehouse Implementation Plan
+
+## Architecture
+
+The solution is organized around a simple event-sourced lakehouse flow:
+
+`source tables -> CDC records -> immutable lake -> current-state warehouse -> catalog`
+
+## Layers
+
+Source:
+- Seven relational e-commerce tables.
+- Strong and weak entities.
+- Primary keys, foreign keys, check constraints, decimal values, timestamps, status domains, and indexes.
+
+CDC:
+- `CDCCapture` records insert, update, and delete events.
+- Each event has a monotonic sequence for ordering and replay.
+- `records_since(offset)` supports checkpoint-style restart.
+
+Lake:
+- `lake_cdc_events` is append-oriented and stores every change.
+- The lake is the authoritative historical record.
+- Duplicate retry appends are skipped by sequence.
+
+Warehouse:
+- Warehouse tables expose latest/current state only.
+- `_cdc_seq` records the latest event applied to each row.
+- `_deleted` preserves source deletes as soft deletes.
+- Restore clears current-state tables and replays lake events up to a target sequence.
+- Point-in-time reads replay lake events up to a target timestamp.
+
+Validation:
+- Source constraints are mirrored by warehouse/data-quality checks.
+- Checks cover keys, referential integrity, enum domains, monetary rules, totals, and date-order rules.
+
+Catalog:
+- `catalog/catalog.json` publishes lake and warehouse datasets with schema, purpose, ownership, consumers, update cadence, lineage, and validation rules.
+
+## Reliability Behaviors
+
+- Inserts, updates, and deletes are captured.
+- Lake retains the full CDC history.
+- Warehouse reflects current state after applying CDC records.
+- Replays after checkpoint use CDC sequence ordering.
+- Duplicate lake writes from retries are skipped.
+- Schema contract failures stop ingestion before downstream application.
+- Historical recovery is performed from the lake, not from separate SCD2 tables.
+
+## Known Production Extensions
+
+- Replace simulated CDC with Debezium/Postgres WAL or equivalent.
+- Use durable cloud/object storage with a table format such as Delta, Iceberg, or Hudi.
+- Store CDC checkpoint state outside the process.
+- Extend schema contracts to validate types, nullability, keys, and enum domains.
+- Add monitoring, alerting, PII controls, and lineage tooling.