DocSync is a data pipeline that ingests documents from multiple heterogeneous sources, normalizes them into a unified PostgreSQL schema, and makes them semantically searchable through a REST API and an MCP tool - queryable directly through natural language in Cursor or Claude Desktop.
Organizations store documents in silos - typed PDFs in one place, scanned contracts in another, structured exports from internal tools somewhere else. Searching across them means manually hunting through three systems, three formats, with no unified way to query semantically.
DocSync solves this by ingesting everything into one place, normalizing it into one schema, and making it searchable with a single natural language query.
Raw Sources
├── Digital PDFs → pdfplumber extracts text layer
├── Scanned PDFs → Azure Document Intelligence OCR (chunked)
└── JSON exports → structured records from internal systems
↓
Unified documents table (PostgreSQL + pgvector)
├── source_type, source_id, raw_text
├── metadata (JSONB - flexible per source)
├── content_hash (MD5 - idempotency)
└── embedding vector(384) - all-MiniLM-L6-v2
↓
FastAPI /search endpoint
├── embed query → cosine similarity → ranked results
└── returns source_type so you know where each result came from
↓
MCP Server
└── search_documents() and get_ingestion_status() as tools
callable from Cursor or Claude Desktop
Keeping vectors inside Postgres means raw text, metadata, and embeddings live in one place. No syncing between two systems, no separate vector store to maintain, and SQL queries work directly alongside semantic search. For a project where the unified schema is the core value, splitting storage would have defeated the purpose.
Azure Document Intelligence was chosen over local OCR (Tesseract, EasyOCR) for two reasons: hands on experience with a managed AI service, and significantly better accuracy on typed scanned documents. The F0 free tier limits each API request to 2 pages, so a pre-processing layer splits multi-page PDFs into 2-page chunks, sends each chunk separately, and stitches results back together - the same pattern you'd use in production when working around any API size constraint.
Each source type is isolated behind its own connector file. Swapping Azure Document Intelligence for Google Cloud Vision (permanent free tier, no per-request page cap) requires changing only scanned_pdf_connector.py - the schema, search endpoint, and MCP server are completely unaffected. This shows separation of concerns across the ingestion layer.
Three reasons Postgres was chosen over MySQL: explicit conflict handling (ON CONFLICT (column) DO UPDATE vs MySQL's less precise ON DUPLICATE KEY UPDATE), native JSONB support for querying inside flexible metadata fields, and pgvector extension support for storing and searching embeddings natively.
- Unified schema - digital PDFs, scanned PDFs, and JSON exports normalized into one
documentstable - Idempotent ingestion - MD5 content hash +
ON CONFLICT DO NOTHINGprevents duplicate processing - Incremental sync -
ON CONFLICT (source_type, source_id) DO UPDATEreprocesses only changed documents - Azure OCR with chunking - works around F0 free-tier 2-page limit for any document size
- Local embeddings -
all-MiniLM-L6-v2via sentence-transformers, no API key, runs on CPU - Semantic search - cosine similarity via pgvector
<=>operator across all source types - MCP server -
search_documentsandget_ingestion_statustools callable from Cursor or Claude Desktop
| Layer | Technology |
|---|---|
| Database | PostgreSQL 16 + pgvector |
| Containerization | Docker + Docker Compose |
| Digital PDF extraction | pdfplumber |
| Scanned PDF OCR | Azure Document Intelligence (F0) |
| Embeddings | sentence-transformers (all-MiniLM-L6-v2) |
| API | FastAPI + uvicorn |
| MCP server | MCP Python SDK (FastMCP) |
| DB connector | psycopg2 |
docsync/
├── connectors/
│ ├── pdf_connector.py # digital PDFs via pdfplumber
│ ├── scanned_pdf_connector.py # scanned PDFs via Azure + chunking
│ └── json_connector.py # structured JSON exports
├── samples/
│ ├── digital/ # digital PDF samples
│ ├── scanned/ # scanned PDF/image samples
│ └── json/ # JSON export samples
├── main.py # FastAPI search endpoint
├── mcp_server.py # MCP server with search tools
├── generate_embeddings.py # embedding generation script
├── run_pipeline.py # master runner for all connectors
├── init.sql # database schema
├── docker-compose.yml # Postgres + pgvector container
├── .env.example # environment variable template
└── requirements.txt
Prerequisites: Docker Desktop, Python 3.11+
# Clone the repo
git clone https://github.com/Lahari-V03/docsync.git
cd docsync
# Create virtual environment
python -m venv venv
venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt
# Copy and fill in environment variables
cp .env.example .env
# Start Postgres
docker compose up -d
# Run ingestion pipeline
python run_pipeline.py
# Generate embeddings
python generate_embeddings.py
# Start search API
uvicorn main:app --reloadSearch via API:
curl -X POST "http://127.0.0.1:8000/search" \
-H "Content-Type: application/json" \
-d '{"query": "LLM guardrails in production", "limit": 5}'Search via MCP (Cursor):
Add to .cursor/mcp.json:
{
"mcpServers": {
"docsync": {
"command": "<path-to-venv>/python.exe",
"args": ["<path-to-project>/mcp_server.py"]
}
}
}| Tool | Description |
|---|---|
search_documents(query, limit) |
Semantic search across all ingested documents |
get_ingestion_status() |
Returns document counts per source type |
DB_HOST=127.0.0.1
DB_PORT=5432
DB_NAME=docsync
DB_USER=your_db_user
DB_PASSWORD=your_db_password
AZURE_DI_KEY=your_azure_key
AZURE_DI_ENDPOINT=your_azure_endpoint- Frontend interface - user uploads documents, searches via UI, receives ranked results
- Google Cloud Vision swap - replace Azure connector with Google Vision for permanent free tier and no per-request page cap (one file change)
- File watcher - auto-trigger ingestion when new files are added to source folders
- POST /ingest endpoint - trigger pipeline via HTTP rather than running scripts manually
- Additional connectors -
.docx(python-docx),.xlsx(openpyxl), web URLs