Implement data-assignment for ecommerce pipeline

[shramore58-byte]

Summary

Implemented a minimal CDC pipeline capturing changes from a MySQL E-commerce source database. The pipeline ingests outbox events into a Parquet data lake and models the current state in a DuckDB warehouse, including schema contract gating, data quality checks, catalog generation, and point-in-time lake restore capabilities.

Source Schema Design

• Domain chosen: E-commerce
• Strong entities: Customers, Products, Orders
• Weak entities: Order Items, Payments, Order Status Events
• Keys, relationships, and indexes: Standard PK/FK relationships (e.g., customer_id on orders, order_id on order_items). Includes indexes on frequent lookup keys.
• Source validation rules: Enforced primary keys and foreign key constraints in schema.sql.

CDC Strategy

• How changes are captured: Trigger-based outbox pattern (I/U/D events inserted into a cdc_events table).
• How inserts, updates, and deletes are handled: Captured with op_type (I, U, D) and a JSON payload in the outbox. Processed into append-only Parquet events in the lake.
• How replay/restart works: Event ID watermarking stored in state/checkpoint.json allowing resumption from the last successfully processed event.
• How duplicates are handled: Checkpoints ensure exactly-once ingestion; DuckDB merge strategies (ON CONFLICT DO UPDATE) ensure idempotency on the warehouse side.

Lake and Warehouse Modeling

• How the lake captures every change: Append-only Parquet files partitioned by source_table and dt (date).
• How the warehouse maintains latest snapshot: DuckDB merges new lake events applying UPSERT for I/U and DELETE for D operations based on primary keys.
• How time travel / restore is supported: Rebuilding warehouse state by replaying lake events up to a specific timestamp using scripts/restore_warehouse.py.

Schema Change Safety

• How incompatible changes are detected: scripts/check_schema_contracts.py compares live MySQL INFORMATION_SCHEMA against a predefined schema_contract_spec.json and a compiled hash (schema_fingerprint.json).
• How ingestion is stopped: Hard stop on mismatch, ingestion exits non-zero and writes to ingest/FAILED_SCHEMA_CHANGE.md.
• How warnings/failures are surfaced: Exits non-zero, creating a clear pipeline failure that alerts engineering.

Validation Parity

• Which source system validations were mirrored downstream: Primary key uniqueness, referential integrity, and not-null constraints.
• How failures are checked and reported: scripts/run_data_quality_checks.py runs against DuckDB to assert these rules on the modeled warehouse data.

Catalog Exposure

• Explain how lake and warehouse datasets are published and discovered: catalog/catalog.json maintains metadata, ownership, and dataset locations for both raw lake partitions and warehouse snapshot tables. Verified using scripts/validate_catalog.py.

Validation

Commands and workflows ran to validate the solution:

docker compose up -d
python -m pipeline.ingest
pytest -q
python scripts/run_data_quality_checks.py
python scripts/validate_catalog.py
python scripts/restore_warehouse.py

Known Limitations / Next Steps

• Production Scale: Migrate from trigger-based outbox to a proper binlog CDC reader (like Debezium or Fivetran).
• Target Systems: Swap DuckDB for Snowflake/BigQuery and the Parquet lake for Iceberg/Delta tables.
• Orchestration: Introduce Airflow/Dagster for orchestration and scheduling instead of manual bash scripts.

Responsible AI Usage

• Did you use AI tools?: Yes, AI was used for codebase scaffolding and testing.
• Where did they help?: Speeding up boilerplate for the pipeline setup, generating schema contracts, and creating unit tests.
• What did you personally verify or correct?: Verified schema correctness, validated the local Docker workflow, tested edge cases of CDC updates and deletes, and reviewed all generated tests for accuracy.

Author Checklist

• Linting passes
• Tests pass
• Model validation/tests pass
• Schema compatibility checks pass
• Data quality validations pass
• Catalog metadata validation passes
• README was tested from a clean setup
• End-to-end CDC flow was validated locally