Privacy-first document intelligence engine for production RAG pipelines.

Parse PDFs, DOCX, PPTX, XLSX & CSV → validated, AI-ready chunks with HITL review.

What is LongParser?

LongParser is an open-source Python library that turns complex documents into clean, structured data that AI systems can use. It reads PDF, DOCX, PPTX, XLSX and CSV files, preserves their structure (headings, tables, equations, page references), and splits them into chunks ready for search and Retrieval-Augmented Generation (RAG).

Most document parsers stop at extracting text. LongParser also covers the steps that follow:

1. Extract: read the document with Docling (the default), or with the optional PyMuPDF or Marker backends.
2. Validate: score chunk quality and redact personal data (PII).
3. Review: let a human approve or edit blocks and chunks before they are indexed (HITL review).
4. Chunk: split the content by token budget, heading hierarchy and table boundaries, with optional semantic boundaries.
5. Embed and index: store chunks in Chroma, FAISS or Qdrant, and query them through a built-in chat engine.

---

Features

Feature	Detail
Multi-format extraction	PDF, DOCX, PPTX, XLSX, CSV via Docling & Marker
Hybrid chunking	Token-aware, heading-hierarchy-aware, table-aware
Semantic chunking	Embedding-based boundaries usingall-MiniLM-L6-v2
Cross-referencing	Deterministic linking of explicit and implicit charts/figures
Quality scoring	Zero-ML heuristic scoring with dictionary & fastText validation
PII redaction	Hybrid Regex + NER (spaCy) redaction with secure HITL preservation
Summary chunks	Async ARQ worker generating hierarchical LLM section summaries
HITL review	Human-in-the-Loop block & chunk editing before embedding
LangGraph HITL	approve / edit / reject workflow with LangGraph interrupt() and MongoDB checkpointer
3-layer memory	Short-term turns + rolling summary + long-term facts
Multi-provider LLM	OpenAI, Gemini, Groq, OpenRouter
Multi-backend vectors	Chroma, FAISS, Qdrant
Production-ready API	FastAPI + Motor (MongoDB) + ARQ + Redis (Queue & Rate Limiting)
Enterprise Security	Tenant isolation, Role-Based Access Control (RBAC), and CORS
LangChain adapters	Drop-inBaseRetriever and LlamaIndex QueryEngine
Privacy-first	All processing runs locally; no data leaves your infra

---

Installation

Quick install (recommended)

pip install "longparser[gpu]"

Includes everything — server, embeddings, vector DB, OCR, LangChain, LlamaIndex. Works on CPU machines too; torch just runs in CPU mode automatically.

Core SDK only (no server, no torch)

pip install longparser

Pick only what you need

Extra	What it adds
server	FastAPI + MongoDB + Redis + LangChain chat
embeddings-gpu	sentence-transformers (GPU)
embeddings-cpu	sentence-transformers (CPU-only torch)
faiss-gpu	FAISS GPU vector store
faiss-cpu	FAISS CPU vector store
chroma	ChromaDB
qdrant	Qdrant
latex-ocr-gpu	pix2tex equation OCR (GPU)
latex-ocr-cpu	pix2tex equation OCR (CPU)
langchain	LangChain core adapter
llamaindex	LlamaIndex reader adapter
gpu	All of the above — one command
cpu	All of the above — CPU-only torch

Advanced: CPU-only install (save ~1.8 GB)

For Docker images, edge devices, or CI environments where CUDA isn't needed:

# Step 1 — CPU torch (~230 MB vs ~2 GB for CUDA)
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

# Step 2 — LongParser CPU bundle
pip install "longparser[cpu]"

---

Quick Start

Python SDK

from longparser import DocumentPipeline, ProcessingConfig

pipeline = DocumentPipeline(ProcessingConfig())
result = pipeline.process_file("document.pdf")

print(f"Pages: {result.document.metadata.total_pages}")
print(f"Chunks: {len(result.chunks)}")
print(result.chunks[0].text)

REST API

# 1. Copy and edit configuration
cp .env.example .env

# 2. Start services (MongoDB + Redis)
docker-compose up -d mongo redis

# 3. Start the API
uv run uvicorn longparser.server.app:app --reload --port 8000

# 4. Upload a document
curl -X POST http://localhost:8000/jobs \
  -H "X-API-Key: your-key" \
  -F "file=@document.pdf"

# 5. Check job status
curl http://localhost:8000/jobs/{job_id} -H "X-API-Key: your-key"

# 6. Finalize and embed
curl -X POST http://localhost:8000/jobs/{job_id}/finalize \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"finalize_policy": "approve_all_pending"}'

curl -X POST http://localhost:8000/jobs/{job_id}/embed \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"provider": "huggingface", "model": "BAAI/bge-base-en-v1.5", "vector_db": "chroma"}'

# 7. Chat with the document
curl -X POST http://localhost:8000/chat/sessions \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"job_id": "your-job-id"}'

curl -X POST http://localhost:8000/chat \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"session_id": "...", "job_id": "...", "question": "What is the refund policy?"}'

---

Architecture

Document → Extract → Validate → HITL Review → Chunk → Embed → Index
                                                              ↓
                                             Chat → RAG → LLM → Answer

Pipeline Stages

1. Extract — Docling converts PDF/DOCX/etc. into structured Block objects
2. Validate — Per-page confidence scoring and RTL detection
3. HITL Review — Human approves/edits/rejects blocks and chunks via the API
4. Chunk — HybridChunker builds token-aware RAG chunks with section hierarchy
5. Embed — Embedding engine (HuggingFace / OpenAI) vectors stored in Chroma/FAISS/Qdrant
6. Chat — LCEL chain with 3-layer memory and citation validation

---

Project Structure

src/longparser/
├── schemas.py           ← core Pydantic models (Document, Block, Chunk, …)
├── extractors/          ← Docling, LaTeX OCR backends
├── chunkers/            ← HybridChunker
├── pipeline/            ← DocumentPipeline
├── integrations/        ← LangChain loader & LlamaIndex reader
├── utils/               ← shared helpers (RTL detection, …)
└── server/              ← REST API layer
    ├── app.py           ← FastAPI application (all routes)
    ├── db.py            ← Motor async MongoDB
    ├── queue.py         ← ARQ/Redis job queue
    ├── worker.py        ← ARQ background worker
    ├── embeddings.py    ← HuggingFace / OpenAI embedding engine
    ├── vectorstores.py  ← Chroma / FAISS / Qdrant adapters
    └── chat/            ← RAG chat engine
        ├── engine.py    ← ChatEngine (LCEL + 3-layer memory)
        ├── graph.py     ← LangGraph HITL workflow
        ├── schemas.py   ← chat Pydantic models
        ├── retriever.py ← LangChain BaseRetriever adapter
        ├── llm_chain.py ← multi-provider LLM factory
        └── callbacks.py ← observability callbacks

---

LangChain Integration

from longparser.integrations.langchain import LongParserLoader

loader = LongParserLoader("report.pdf")
docs = loader.load()  # list[langchain_core.documents.Document]

LlamaIndex Integration

from longparser.integrations.llamaindex import LongParserReader

reader = LongParserReader()
docs = reader.load_data("report.pdf")

---

Configuration

Copy .env.example to .env and set:

Variable	Default	Description
LONGPARSER_MONGO_URL	mongodb://localhost:27017	MongoDB connection
LONGPARSER_REDIS_URL	redis://localhost:6379	Redis for job queue & rate limits
LONGPARSER_LLM_PROVIDER	openai	LLM provider
LONGPARSER_LLM_MODEL	gpt-5.3	Model name
LONGPARSER_EMBED_PROVIDER	huggingface	Embedding provider
LONGPARSER_VECTOR_DB	chroma	Vector store backend
LONGPARSER_CORS_ORIGINS	*	Allowed CORS origins
LONGPARSER_RATE_LIMIT	60	Max RPM per tenant
LONGPARSER_ADMIN_KEYS	(empty)	Comma-separated admin API keys

---

Running with Docker

docker-compose up

API available at http://localhost:8000 · Docs at http://localhost:8000/docs

---

Testing

# Install dev dependencies
uv sync --extra dev

# Run unit tests
uv run pytest tests/unit/ -v

# Run with coverage
uv run pytest tests/ --cov=src/longparser --cov-report=term-missing

---

Part of the Long Suite

LongParser is part of the Long Suite — a collection of tools for building, testing, and monitoring production RAG systems.

Project	Description
LongParser	Document ingestion and chunking (you are here)
LongTrainer	RAG chatbot framework
LongGuard	In-flight circuit breaker and reasoning loop recovery for agents
LongCache	Semantic similarity cache for LLM and RAG pipelines
LongTracer	Hallucination detection and tracing
LongProbe	Retrieval regression testing

---

Contributing

See CONTRIBUTING.md for development setup and PR guidelines.

Security

See SECURITY.md for vulnerability reporting.

License

MIT — Copyright © 2026 ENDEVSOLS