RAG Document Q&A API

An independent Retrieval-Augmented Generation prototype for querying PDF documents using natural language. It combines retrieval, reranking, optional tracing and an evaluation harness.

Status: local portfolio project. Authentication, tenant separation and published quality/latency baselines remain roadmap items. Use sample or non-sensitive documents in a trusted local environment; the repository does not establish production deployment or measured accuracy.

What it does

1. Upload PDF documents via web UI or API
2. Documents are chunked and embedded using sentence-transformers (local, free)
3. Embeddings stored in ChromaDB (local vector database)
4. Ask questions in natural language → hybrid retrieval (BM25 + vector + cross-encoder reranking) → LLM generates answer with source citations
5. Optional Langfuse tracing records retrieval and generation spans when configured
6. A RAGAS evaluation harness supports quality measurement; baseline results have not yet been published

Stack

• FastAPI — REST API + web UI
• LangChain — RAG orchestration, document chunking
• ChromaDB — vector database (local, persistent)
• sentence-transformers — local embeddings + cross-encoder reranker (no API key needed)
• rank-bm25 — keyword search for hybrid retrieval
• OpenRouter / OpenAI — LLM for answer generation (configurable)
• Langfuse — observability and tracing (optional, self-hosted)
• RAGAS — evaluation metrics (dev dependency)

Key features

Hybrid search + reranking

Pure vector search misses exact terms (proper nouns, error codes, numbers). The hybrid retriever combines:

• BM25 keyword search (in-memory, synced with ChromaDB)
• Vector search (ChromaDB cosine similarity)
• Reciprocal Rank Fusion (RRF) to merge rankings without comparable score scales
• Cross-encoder reranking (cross-encoder/ms-marco-MiniLM-L-6-v2) to reorder the final candidates

Toggle with HYBRID_SEARCH=true and RERANKER_ENABLED=true.

Observability (Langfuse)

When Langfuse is configured, /api/ask instrumentation records:

• A retrieval span (query, chunks, scores, latency, mode)
• A generation span (model, prompt, answer, token usage, latency)

No-op when Langfuse keys are not set — the app runs identically without an observability backend.

Evaluation (RAGAS)

Use the evaluation harness to measure a documented dataset and configuration. The presence of the harness alone is not evidence of answer quality:

pip install -r requirements-dev.txt
# Upload documents, then:
python -m eval.run_eval --dataset eval/golden_dataset.json --k 5

Metrics: faithfulness, answer relevancy, context precision, context recall.

The golden dataset (eval/golden_dataset.json) is versioned JSON — run before/after any retrieval change and compare.

CI/CD

GitHub Actions runs ruff check + pytest on every push and PR (.github/workflows/ci.yml).

Roadmap

The following roadmap separates existing components from planned work. Published evaluation reports are still needed to establish quality and latency.

Phase 0 — Retrieval prototype components

• Hybrid search (BM25 + vector + RRF fusion) with cross-encoder reranking
• Optional Langfuse tracing (retrieval + generation spans)
• RAGAS evaluation harness with versioned golden dataset
• CI: lint + tests on every push

🚧 Phase 1 — Scale & reliability (in progress)

• Async ingestion: Redis-backed task queue (arq) for PDF processing — job status endpoint, retries, backpressure. Upload returns 202 + job_id instead of blocking.
• Feedback loop: POST /api/feedback (👍/👎 per answer) stored in Langfuse → dataset for prompt/retrieval tuning.
• One-command stack: docker compose up brings up app + Redis + Langfuse.
• Baseline metrics published: RAGAS scores + p95 latency documented in this README.

Phase 2 — Multi-user & guardrails

• Collections / multi-tenancy: namespaced document sets per user or project (ChromaDB collections) with per-collection queries.
• Groundedness guardrails: out-of-domain question detection ("not enough context") and citation enforcement — answers must reference retrieved chunks.
• Auth: API-key per tenant, rate limiting.

Phase 3 — Continuous improvement

• Prompt/retrieval A-B testing driven by feedback + RAGAS regression in CI (eval must not drop vs. baseline).
• Query analytics dashboard: top questions, failure clusters, thumbs-down rate over time (Langfuse + Grafana).
• Cost/latency optimization: semantic caching, embedding quantization, model routing per query complexity.

Before claiming a deployment milestone, publish a reproducible evaluation report and verify the relevant access controls and failure handling.

Quick start

# 1. Install dependencies
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# 2. Configure LLM credentials for generated answers (embeddings run locally)
cp .env.example .env
# Edit .env with your OpenRouter API key

# 3. Run
uvicorn main:app --reload --port 8000

# 4. Open http://localhost:8000

Docker

docker compose up --build

The API will be available at http://localhost:8000.

API endpoints

Method	Path	Description
POST	/api/upload	Upload PDF document(s)
POST	/api/ask	Ask a question (returns answer + sources)
GET	/api/documents	List uploaded documents
DELETE	/api/documents/{id}	Delete a document
GET	/api/health	Health check (includes hybrid + tracing status)

Architecture

[PDF Upload] → [PyPDF Loader] → [Text Splitter] → [Embeddings]
                                                        ↓
                                                 [ChromaDB]
                                                        ↓
[User Question] → [BM25 Search] ──┐                    │
              → [Vector Search] ─┤                    │
                                   ↓                    │
                          [Reciprocal Rank Fusion] ←────┘
                                   ↓
                         [Cross-Encoder Reranker]
                                   ↓
                          [Top-K Chunks]
                                   ↓
                          [LLM Prompt + Context]
                                   ↓
                          [Answer + Sources]
                                   ↓
                        [Langfuse Trace]

Development

# Install dev dependencies
pip install -r requirements-dev.txt

# Run tests
pytest

# Lint
ruff check .

# Evaluate pipeline quality (requires OPENROUTER_API_KEY + uploaded docs)
python -m eval.run_eval

Environment variables

See .env.example for all configurable options.

Author

Ray Peratta — GitHub — LinkedIn