debezium/dbz#2229 feat: implement batch compaction, sync_batch, and T…

[KMohnishM]

…hreadPoolExecutor parallelization

Description

This PR completes the GSoC Week 8 milestone by adding batch orchestration, CDC event compaction, and concurrency to the SyncManager pipeline. It optimizes the consumption of multiple events by aggregating redundant transactions on identical primary keys and parallelizing document construction.

Proposed Changes

• pydebeziumai.sync.manager:
  • Implemented sync_batch(self, events: list[DebeziumEventModel]) in SyncManager.
  • Implemented a CDC Compaction algorithm that groups batch events by doc_id and retains only the latest operation state (e.g., c + u + d becomes a single d event; c + u becomes a single u event). This prevents redundant updates and deletes from reaching the vector store.
  • Added ThreadPoolExecutor concurrency during the document building phase to calculate embeddings and build LangChain Documents in parallel.
  • Added a sequential fallback error-handler routine: if a batch vector operation fails, it falls back to processing events one-by-one to isolate the failure and route the specific offending event to the Dead Letter Queue (DLQ).
• tests/unit/test_sync_manager:
  • Added test_sync_batch_compaction verifying stateful CDC event collapse behavior.
  • Added test_sync_batch_error_fallback verifying sequential fallback isolation during bulk adapter failures.

Verification Results

All 88 unit and integration tests passed successfully: