Try It Live →  ·  View on GitHub

The buttons shown inside the GIF above are part of the mockup graphic, not a live page: use the real links directly above instead.

A grounded retrieval-augmented generation system that started as a general-purpose RAG pipeline and grew into a self-evaluating knowledge engine: it answers questions, ranks live-scraped job leads, and drafts cover letters: all sourced only from a private corpus, with citations, never invented. It detects when its own sources disagree or have gone stale, learns from queries it initially failed to answer, and asks for clarification instead of guessing when it genuinely doesn't know.

Try it live: /docs is a real, interactive Swagger UI for the deployed API: click "Try it out" on any endpoint to run it against a live, isolated demo corpus (real public job postings only, never the private one). No custom frontend deployed yet, but the API itself is real and running: curl examples below work the same way.

---

Tech Stack

🔌 AI & Protocols	💻 Languages

🌐 Web & Backend Frameworks	☁️ Cloud & Hosting

🗄️ Databases & Vector Search	🧰 Dev Tools

---

Architecture

Query
  → Intent classification (LangGraph agent: answer / rank / draft)
  → answer/draft: Hybrid Retrieval (BM25 sparse + real semantic vector search, RRF fusion;
                  recency-weighted + freshness-flagged, near-duplicate chunks collapsed,
                  hierarchical document-then-chunk ranking, personalized toward active-focus
                  topics, boosted by a knowledge graph of known entities and by sources that
                  previously resolved similar failed queries)
                  → Reranking (Cohere API, or a local embedding-similarity fallback)
  → rank: loads the scored-leads corpus directly (a vague query like
          "rank my leads" shares almost no vocabulary with the lead
          documents, so similarity search reliably misses it)
  → Claude generation, grounded strictly to retrieved context, flags source contradictions inline
  → Citation verification (existence + semantic support + staleness per claim)
  → Confidence state (high / medium / insufficient / conflicting / stale)
  → Knowledge-state agent decides: answer, or (if still insufficient after a widened
    retry) ask a grounded clarifying question instead of guessing
  → Answer + citations + confidence + conflicts + action

Real hybrid retrieval, not a lexical trick wearing a "dense" label. BM25 sparse scoring fused (RRF) with genuine semantic search: local embeddings (fastembed, ONNX, no torch, no API key) indexed in a real persistent vector database (Chroma, embedded, no server to run). A process restart on an unchanged corpus loads the existing index instead of re-embedding from scratch. No Pinecone required to run it (documented but unimplemented). Generation is a dependency-free urllib call to an Anthropic Messages API-compatible endpoint (configurable base URL/model, so it can be pointed at a proxy or alternate provider without touching code). The one real framework dependency is LangGraph, used for the agent's intent-routing state machine.

Why hybrid retrieval, and why a real agent instead of one Q&A endpoint

Stock LLMs hallucinate on private data. This system grounds every answer in retrieved source passages and cites them, but the more interesting problem it solves is that not every query is a similarity search. "Rank my top 3 leads" shares almost no vocabulary with lead documents (which contain Company: / Score: 87/100, not the word "rank"), so lexical or dense retrieval alone reliably misses it. The LangGraph agent classifies intent first and takes a different path for rank (load the scored-leads corpus directly) versus answer/draft (real retrieval + reranking): a real bug this caught and fixed, not a theoretical design choice.

Features

Core retrieval

• Hybrid retrieval: BM25 sparse scoring + real local semantic embeddings in a persistent Chroma vector DB, fused via RRF, near-duplicate chunks collapsed before the final cut
• Hierarchical (document-then-chunk) ranking: ranks at the document level first, then only lets chunks from the top-ranked documents compete for the final cut, so a document with several genuinely relevant chunks wins over an isolated one-off match elsewhere
• Personalized retrieval: an optional, hand-edited list of "current focus" topics boosts matching content, on top of the same multiplier chain as recency/authority scoring
• Knowledge graph retrieval boost: a lightweight document-entity graph built from real cross-references in the corpus; a document connected to whatever the query mentions gets boosted, even if it scores poorly on raw text similarity
• Learning from failed searches: when a query only resolves after a widened retry, the source(s) that actually answered it are remembered and boosted for similar future queries
• LangGraph agent: classifies each query as answer, rank, or draft and routes accordingly, rather than forcing every request through one generic RAG path

Trust & verification

• Citation verification: every [Source: X] claim is checked two ways: did the model cite a source that actually exists in the retrieved context, and does that source's content actually semantically support the claim (embedding similarity, not just trust)
• Freshness flags: a source older than 90 days gets flagged stale per citation, distinct from (and in addition to) the ranking-time recency boost
• Contradiction detection: when two retrieved sources genuinely disagree, the generator flags it inline and explains which is more current, based on dates in the context; embedding similarity can't do this (contradicting sentences on the same topic sit close together in vector space), so this is generation-based, not a second retrieval pass
• 5-state confidence model: every answer carries high_confidence / medium_confidence / insufficient_evidence / conflicting_evidence / stale_evidence, derived from real signals (rerank score, source count, citation-check results, staleness, detected conflicts): no second LLM call
• Knowledge-state agent: acts on the confidence state instead of only reporting it: if a query is still insufficient after a genuine widened retry, the API returns action: "clarify" and a real clarifying question naming the closest material actually found, instead of a fabricated or flatly unhelpful answer

Evaluation

• Deterministic eval benchmark (/eval/run, personal corpus only): a small hand-curated Q&A set run through the real pipeline end to end, reporting measured retrieval hit rate, retrieval precision, citation support rate, hallucination rate, confidence-state distribution, latency, and real input/output token counts: no LLM judge dependency
• Ragas evaluation (optional, unwired): faithfulness ≥ 0.75, answer relevancy ≥ 0.80; the deterministic benchmark above is what actually runs today

Product surface

• Isolated public demo corpus: a second, completely separate stack (/demo/ask) indexed only from real public job postings (including a self-labeled synthetic conflicting pair used to demo contradiction detection live), so it can be tried without ever exposing the private corpus. Personalization and failed-search learning are deliberately not applied here: no reason to bias what an anonymous visitor sees.
• Cover letter drafting: grounds every claim in the actual candidate background and job details present in the retrieved context; explicitly instructed not to invent employers or details
• Lead ranking: only ranks entries with a real, explicit Score: N/100 field from live-scraped leads, refusing to substitute anything else if none are present
• MCP server (mcp_server.py): exposes the whole agent as a single ask_job_leads tool directly inside Claude Code
• Next.js chat frontend: /chat endpoint wired to a real UI, not just a curl-only API (not yet deployed publicly: see Roadmap)

Quick Start

git clone https://github.com/albatrossflyon-coder/rag-system
cd rag-system
python -m venv venv313
venv313\Scripts\activate      # Windows; source venv313/bin/activate elsewhere
pip install -r requirements.txt
cp .env.example .env           # Add ANTHROPIC_API_KEY at minimum

Drop markdown/JSON documents into data/corpus/, then:

# Start the API server (also powers the MCP server + Next.js frontend)
cd api && python main.py

# Ask a grounded question
curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What does the contract say about payment terms?"}'

# Agent-routed chat (answer / rank / draft, auto-classified)
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"query": "rank my top 3 leads"}'

# Same pipeline, isolated public demo corpus -- never touches the private one
curl -X POST http://localhost:8000/demo/ask \
  -H "Content-Type: application/json" \
  -d '{"query": "What remote software jobs are available?"}'

# Real, measured quality numbers -- not vibes
curl http://localhost:8000/eval/run

# Query without the server (local pipeline)
python query_local.py "What is MCP?" --answer

# Frontend
cd frontend && npm install && npm run dev

Configuration

Variable	Required	Description
ANTHROPIC_API_KEY	Yes	Claude generation + intent classification (an API key for whatever endpoint ANTHROPIC_BASE_URL points at)
ANTHROPIC_BASE_URL	No	Defaults to https://api.anthropic.com/v1/messages. Point at any Anthropic Messages API-compatible endpoint
ANTHROPIC_MODEL	No	Defaults to claude-sonnet-4-6
COHERE_API_KEY	No	Cohere reranking (falls back to local embedding-similarity reranking)
PINECONE_API_KEY	No	Documented, not currently wired: local BM25 + Chroma retrieval is what actually runs
RETRIEVAL_TOP_K / RERANK_TOP_K	No	Defaults: 10 / 5
CHUNK_MAX_TOKENS / CHUNK_OVERLAP	No	Defaults: 1024 / 100
DEMO_CORPUS_PATH	No	Defaults to data/corpus-demo/: the isolated public demo corpus
PUBLIC_DEPLOYMENT	No	When true, the personal corpus/agent is never constructed (not just route-gated) and /ask, /chat, /eval/run, /metrics all 404. Set on the public Fly.io deployment
ACTIVE_FOCUS_TOPICS	No	Comma-separated, e.g. job search,rag-system. Hand-edited as priorities shift, not redeployed: boosts matching content in the personal corpus only. Default: job search,rag-system

Project Structure

rag-system/
├── api/main.py              # FastAPI: /ask, /demo/ask, /chat, /eval/run, /health, /metrics
├── mcp_server.py             # ask_job_leads MCP tool for Claude Code
├── eval_benchmark.py          # Deterministic quality benchmark, no LLM judge
├── ingest_job_hunter_leads.py # Pulls live scored leads from job-hunter into the corpus
├── ingest_vault_notes.py      # Pulls curated career-relevant notes from the Obsidian vault
├── fetch_public_demo_data.py  # Pulls real public postings into the isolated demo corpus
├── ROADMAP.md                 # v2 roadmap -- mapped against commercial RAG platforms + recent research
├── src/
│   ├── agent/graph.py        # LangGraph intent router: answer / rank / draft
│   ├── retrieval/
│   │   ├── hybrid_retriever.py    # BM25 + semantic search, RRF, hierarchical ranking,
│   │   │                          # personalization + graph + feedback boosts
│   │   ├── vector_store.py        # fastembed + persistent Chroma
│   │   ├── knowledge_graph.py     # Lightweight document-entity graph (GraphRAG-lite)
│   │   └── feedback_store.py      # Learning from failed searches (retry-outcome memory)
│   ├── reranking/            # Cohere or local embedding-similarity reranking
│   ├── generation/           # Anthropic Messages API-compatible calls, grounded, cited, conflict-flagged
│   ├── ingestion/             # Document loading (recursive) + semantic chunking
│   ├── verification/          # Citation checks, staleness, contradiction parsing, confidence + action decision
│   └── evaluation/            # Ragas faithfulness + relevance scoring (optional, unwired)
├── frontend/                  # Next.js chat UI
├── data/corpus/                # Private source documents (markdown/JSON)
├── data/corpus-demo/            # Public demo corpus, fully isolated from the private one
└── query_local.py              # CLI query tool, no server needed

Roadmap

Benchmarked against 10 major commercial RAG platforms (Azure AI Search, Bedrock Knowledge Bases, Vertex AI RAG Engine, OpenAI Vector Stores, Pinecone, Elastic, Databricks, Weaviate, LlamaIndex, Cohere) plus recent research (GraphRAG, LightRAG, GRADRAG): see ROADMAP.md for the full comparison and what's next: query rewriting/routing, context compression, true multi-hop retrieval, structured/table/multimodal retrieval, semantic caching, model routing, and permission-aware retrieval.

Author

Chris Brown
Portfolio