debezium/dbz#2128 feat: add PGVector adapter and core logging improve…

[KMohnishM]

Description

This Pull Request implements the first half of GSoC Week 5 targets under tracking issue debezium/dbz#2128. It introduces support for the PGVector (PostgreSQL) vector database backend and establishes a standardized, library-wide logging and observability framework.

Key Changes

1. PGVector Adapter:
   • Implemented PGVectorAdapter in pydebeziumai/adapters/pgvector.py extending the core VectorStoreAdapter contract.
   • Leveraged the langchain-postgres backend underneath to perform idempotent upsert, delete, and retriever generation.
   • Registered PGVectorAdapter in the public adapters interface (pydebeziumai/adapters/__init__.py).
2. Standardized Logging & Observability:
   • Integrated python-native logging throughout all core lifecycle layers.
   • SyncManager: Log all CDC synchronization activities (INFO level for inserts, updates, deletes, and soft-deletes) and clear warnings for retries and Dead Letter Queue (DLQ) routing.
   • DocumentBuilder: Added debug logs tracing document parsing, building, and tombstone discard phases.
   • IdStrategy & ProjectionPolicy: Injected debug logs indicating selected formatting policies and resolved document ID values.
3. Tests:
   • Added a comprehensive unit/mock test suite in tests/unit/test_pgvector_adapter.py validating initialization, correct backend instantiation, mock-upsert, mock-delete, exception fallbacks, and retriever exports.

Verification Results

All code validation checks have been successfully run and passed within WSL:

• Ruff Quality Check: Passed (All checks passed!).
• MyPy Type Safety: Passed (Success: no issues found in 22 source files with strict mode enabled).
• PyTest Execution: 79/79 unit and integration tests passed successfully.

[vjuranek]

Juts one minor comment about logging, otherwise LGTM.

[vjuranek]

IMHO it's better do the logging first and then do the action itself - if anything goes wrong, you know from log what action triggered it (e.g. exception with full stack trace can swallowed somewhere etc)

[jpechane]

@KMohnishM I am a bit confused, the PR contains Miluvs related changes I already merged in the other PR.

[KMohnishM]

Hi @jpechane, apologies for the confusion! Here is what happened with the stacked branches:

1. The Stacked Structure:

   • gsoc-week-5-pgvector (PR debezium/dbz#2128 feat: add PGVector adapter and core logging improve… #11) is the base branch containing PGVector adapter changes.
   • gsoc-week-5-milvus (which contains the Milvus adapter) was stacked on top of it, meaning the Milvus branch naturally inherited the PGVector commits.

2. The Source of the Confusion:

   • During a recent git synchronization, the Milvus commits were pushed down into the gsoc-week-5-pgvector branch, making the Milvus changes show up in this PR (debezium/dbz#2128 feat: add PGVector adapter and core logging improve… #11) too.

3. The Resolution:

   • I have just rebased , cleaned up and force-pushed the gsoc-week-5-pgvector branch.
   • It now contains only the PGVector adapter and the core logging improvements. The Milvus commits have been completely removed from this branch.

Everything should look clean and correct now!

[KMohnishM]

@jpechane One small request is to merge every PR also to main branch as the PRs are stacked there base branch is the sub sequent branch rather than main branch.

[jpechane]

@KMohnishM Thanks for the explanation, I completely forgot about the PR stacking