🧱 TextStruct — Project ToDo & Roadmap

A fully local, layout-aware OCR → structure → RAG pipeline for scanned textbooks.

✅ Phase 0 — Foundation (DONE)

Goal: Stable, portable, local-first OCR pipeline.

✅ Phase 1 — Text Quality & Cleanup (HIGH PRIORITY, DONE)

Goal: Ensure text is clean before structure inference.

Header/footer removal
- Frequency-based detection
- Page-position awareness (top/bottom)
Whitespace normalization
Line reflow & de-hyphenation
Paragraph boundary stabilization
Debug cleanup mode (--debug-clean)
- Before/after page dumps
- Removed-line inspection

✅ Phase 2 — Section & Subsection Detection (VERY HIGH ROI, DONE)

Goal: Move beyond chapters → sections for better RAG granularity.

Heuristic section detection
- Numbering patterns (1.1, 1.2, A.1, etc.)
- Title casing / ALL CAPS detection
- Short-line heuristics
Optional LLM validation for ambiguous sections
sections.json generation
- Section titles
- Page ranges
Alignment with cleaned text pages

🟡 Phase 2.x — Section Refinement (DEFERRED)

Goal: Improve section accuracy in noisy textbooks.

Merge multi-line section titles
Detect consecutive title lines
Filter false positives
- Shape names
- Figure labels
- Single-word headings
Fix invalid page spans
- start_page <= end_page
- Drop zero-length sections
Optional LLM-based structural validation

✅ Phase 3 — Chunk Builder (RAG-Ready Output, DONE)

Goal: Produce structured, retrievable chunks.

Chunk Schema

text
chapter
section
page_start
page_end
subject
grade
book
source

Implementation

✅ Phase 3.x — Chunk Refinement (DONE)

Goal: Improve chunk semantic quality.

Post-chunk refinement hook (--refine)
OCR artifact removal
QR code / printer ID removal
Pedagogical callout filtering
- Figure it Out
- Math Talk
- Think
- Do This
- Activity
Separate exercises / answers from exposition
- Pedagogical content → metadata
- Explanatory text → RAG body
Semantic re-chunking (embeddings)
- Model: all-mpnet-base-v2
- Disk caching (.doctr_cache/embeddings/)
- Merge-focused algorithm
- Output: chunks_semantic.json
- CLI: --semantic-chunk
- 29 unit tests passing

🟢 Phase 3.y — Pedagogical Role Mapping (READY TO IMPLEMENT)

Goal: Assign functional knowledge roles to chunks to enable intent-aware retrieval
(e.g., searching definitions vs. examples vs. procedures).

Architecture

Hybrid two-tier system

Tier 1 (Default): Heuristics + Embedding Similarity
- Covers ~95% of chunks
- <1s per chunk
Tier 2 (Optional): LLM Classification
- Used only for uncertain cases
- Enabled via --classify-roles-llm

Pedagogical Roles (Locked Vocabulary)

Role	Purpose	Example Pattern
`definition`	What X is	“X is defined as…”
`explanation`	Conceptual reasoning	“This happens because…”
`procedure`	Steps / methods	“Step 1… Step 2…”
`example`	Worked illustration	“Example 1.2…”
`question`	Prompts / practice	“Why do you think…?”
`result`	Findings / theorems	“Therefore, we conclude…”
`reference`	Lookup material	Tables, formulas, lists
`evidence`	Data-backed claims	“According to the study…”
`visual_proxy`	Describes visuals	“Figure 2 shows…”
`context`	Background / motivation	“Historically…”

Implementation Breakdown

Phase 1 — Role Reference Library

Create src/role_references.py
Add 3–5 curated examples per role
Pre-embed reference texts at initialization
Cache embeddings on disk

Phase 2 — Heuristic Matching

Create src/role_heuristics.py
Implement regex-based fast classification
High-confidence patterns (≥0.85)
Medium-confidence patterns (0.65–0.84)
Return (None, 0.0) for uncertain cases

Phase 3 — Main Classifier

Phase 4 — LLM Fallback

Create src/role_llm.py
Few-shot prompting (3 examples)
Force strict JSON output
Validate role against locked vocabulary
Low temperature (0.1) for determinism

Phase 5 — Pipeline Integration

Phase 6 — CLI Validation

Require --semantic-chunk
Emit clean, actionable error messages
Prevent invalid flag combinations

Phase 7 — Testing

Create tests/test_role_classifier.py
Unit tests for heuristics
Unit tests for embedding similarity
Batch-processing tests
Full pipeline integration test

Output Schema

{
  "metadata": {
    "pedagogical_role": "definition",
    "role_confidence": 0.92
  }
}

## ✅ Phase 3.z — Deferred Enhancements (PARTIAL)

### 🟢 Phase 3.z1 — Spatial-Semantic Linking (READY TO IMPLEMENT)
**Goal:** Connect text chunks to nearby visuals for richer retrieval.

- [ ] If `pedagogical_role == visual_proxy`, use bounding boxes to locate nearest:
  - Table (using `table_metadata` from Phase 4.x)
  - Figure (using `figure_metadata` from Phase 4.x)
  - Diagram
- [ ] Compute spatial proximity using OCR block geometry
- [ ] Add `linked_visual_id` to chunk metadata
- [ ] Enable downstream Vision-RAG hooks

**Status**: Ready to implement - Phase 4.x provides table and figure detection infrastructure. Can now link text chunks to detected visual elements using spatial proximity.

---

### ✅ Phase 3.z2 — Role Tie-Breaking (DONE)
**Goal:** Resolve ambiguous pedagogical intent.

- [x] Assign `secondary_role` when `has_pedagogical_callout == true`
- [x] Handle mixed-intent chunks (e.g., `definition + exercise`)
- [x] Preserve both primary and secondary role confidence scores
- [x] Improve mixed-intent retrieval

**Implementation**:
- Created `src/role_tiebreaker.py` with `assign_secondary_role()` and `process_tie_breaking()`
- Integrated into pipeline after role classification
- Secondary roles assigned when callout content differs from main content role
- Confidence threshold: 0.60 for secondary role assignment
- 12/12 unit tests passing in `tests/test_phase_3z.py`

**Output Schema**:
```json
{
  "metadata": {
    "pedagogical_role": "definition",
    "role_confidence": 0.92,
    "secondary_role": "question",
    "secondary_role_confidence": 0.85
  }
}

✅ Phase 3.z3 — Metadata Refinement (DONE)

Goal: Fix OCR-induced structural drift.

Implement "Last Known Section" inheritance
Repair orphan chunks caused by page breaks
Stabilize chapter/section continuity
Drop structurally invalid chunks

Implementation:

Created src/metadata_refinery.py with comprehensive refinement functions
repair_orphan_chunks(): Propagates last known section/chapter to orphans
stabilize_section_continuity(): Smooths single-chunk section anomalies
drop_invalid_chunks(): Removes chunks with invalid text length or page ranges
Integrated into pipeline after semantic chunking
Adds metadata flags: section_inherited, chapter_inherited, section_stabilized

Statistics Reported:

Number of invalid chunks dropped
Number of orphan chunks repaired with section inheritance

✅ Phase 3.zx — Math & OCR Cleanup (DONE)

Goal: Make mathematical content searchable, stable, and LLM-safe.

Core Module Implementation

Created src/math_refinery.py with comprehensive cleanup functions
Deterministic regex-based normalization (fast, predictable)
Optional LLM fallback for complex formulas (via Ollama)
Metadata tracking for applied fixes
Batch processing with progress bars

Angle Symbol Normalization

Convert OCR errors Z[A-Z] → ∠[A-Z] (e.g., ZABC → ∠ABC)
Convert OCR errors /[A-Z] → ∠[A-Z] (e.g., /XYZ → ∠XYZ)
Word boundary detection to avoid false positives
Support for 2-3 letter angle patterns

Q./Ans. Dialogue Labeling

Detect Q. / Ans. / Answer. at line starts
Wrap in Markdown:
- Q. → **Q:**
- Ans. → **Ans:**
- Answer. → **Answer:**
Prevent LLM misclassification as narrative text

Word Break Repair

Fix common OCR segmentation errors:
- ifit → if it
- ofthe → of the
- inthe → in the
- tothe → to the
- isthe → is the
- andthe → and the

Math Operator Spacing

Normalize spacing around operators:
- AB=CD → AB = CD
- A+B → A + B
- A-B → A - B
Pattern-based spacing for word-character operators

Optional Ollama-Powered Math Fix

Flag-based activation (--fix-math-ollama)
LLM cleanup for complex formulas (slow)
Automatic detection of complex math content
Graceful fallback on LLM failure
HTTP API integration with Ollama

Testing & Integration

33 comprehensive unit tests in tests/test_math_refinery.py
All tests passing in Docker environment
Integrated into src/refine.py pipeline
Integrated into src/pipeline.py for semantic chunks
CLI flags --fix-math and --fix-math-ollama
Validation: --fix-math-ollama requires --fix-math

Output Schema

{
  "text": "**Q:** Find the measure of ∠ABC given that AB = 5 cm.",
  "metadata": {
    "math_fixes_applied": 3
  }
}

Deferred Features

Formula integrity validation (balanced parentheses, etc.)
LaTeX conversion for complex formulas
Math expression simplification
Unit conversion (cm → mm, etc.)(Technically not a necissity,we could let the llm just fix it)
Advanced sequence repair beyond operator spacing

✅ Phase 4 — Layout-Aware Processing (DONE)

Goal: Correctly process multi-column and complex page layouts.

Implementation Complete

Architecture

Algorithm: X-histogram clustering for column detection
- Builds histogram of block x-centers across page width (100 bins)
- Smooths with moving average (window=5)
- Finds valleys (local minima) as column boundaries
- Assigns blocks to columns and sorts left-to-right
- Within columns: sorts top-to-bottom, then left-to-right
Integration: Reading order resolved AFTER header removal, BEFORE text extraction
Performance: O(n) complexity, <10ms per page with 50 blocks

CLI Usage

# Enable layout-aware processing
python src/main.py input.pdf --layout-aware

# Add visual debugging
python src/main.py input.pdf --layout-aware --debug-layout

✅ Phase 4.x — Advanced Layout Features (DONE)

Goal: Extend layout-aware processing with advanced structural detection.

Implementation Complete

Paragraph/block grouping via spatial cues
- Vertical gap analysis with adaptive thresholding
- Dynamic threshold = minimum_gap × multiplier (default: 1.5)
- O(n log n) performance, <5ms per page
- Module: src/layout_grouping.py
- Tests: 11 unit tests passing in tests/test_layout_grouping.py
Table detection (grid structures)
- Simple mode: Alignment-based grid detection (min 3 rows × 2 cols)
- Advanced mode: Adds density + whitespace uniformity analysis
- DBSCAN-like spatial clustering for dense regions
- Row/column clustering with alignment scoring
- Grid score threshold: 0.60 (simple), 0.65 (advanced)
- O(n²) simple mode, O(n² log n) advanced mode
- Module: src/layout_table_detection.py
- Tests: 14 unit tests passing in tests/test_table_detection.py
Figure caption linking (spatial proximity)
- Regex-based caption detection ("Figure X:", "Fig. X:", "Table X:", etc.)
- Marks caption blocks with metadata
- Simplified implementation focusing on caption identification
- Module: src/layout_figure_linking.py
Sidebar handling (callouts as secondary content)
- Detects narrow columns (<35% page width) in margins
- Position detection: left margin, right margin, or embedded
- Marks blocks with sidebar metadata
- Module extensions: src/layout_order.py

Data Model Extensions

Extended OCRBlock dataclass with Phase 4.x metadata fields:

paragraph_id: Optional[int] - Sequential paragraph identifier
table_metadata: Optional[Dict[str, Any]] - Table detection results
- Schema: {is_table, table_id, detection_mode, grid_score, rows, cols, confidence}
figure_metadata: Optional[Dict[str, Any]] - Figure/caption information
- Schema: {is_figure, figure_id, is_caption, figure_number, has_caption, proximity_score}
sidebar_metadata: Optional[Dict[str, Any]] - Sidebar classification
- Schema: {is_sidebar, sidebar_type, width_ratio, position}

Added helper properties:

width, height, area for spatial calculations

Pipeline Integration

Processing order (inserted after header/footer removal):

Sidebar detection (BEFORE reading order)
Reading order resolution
Paragraph grouping (always run with Phase 4.x features)
Table detection (if --detect-tables)
Figure linking (if --detect-figures)

CLI Flags

# Individual feature flags
python src/main.py input.pdf --layout-aware --detect-tables
python src/main.py input.pdf --layout-aware --detect-figures
python src/main.py input.pdf --layout-aware --detect-sidebars

# Enable all Phase 4.x features at once
python src/main.py input.pdf --layout-aware --advanced-layout

Flag validation: Phase 4.x features require --layout-aware

Testing

25+ unit tests across all features
All tests passing in Docker environment
Coverage: edge cases, integration scenarios, helper functions

Deferred Features

Figure region detection (whitespace analysis)
Proximity linking between figures and captions
GPU-accelerated layout operations
Visual element extraction for vision-RAG

✅ Phase 5 — Embeddings & RAG Integration (DONE)

Goal: Enable retrieval and downstream QA.

Embedding backend interface
- Local models only (sentence-transformers)
Embed chunks.json / chunks_semantic.json
Pluggable vector store
- InMemoryVectorStore with cosine similarity search
- Persistent storage via pickle
Retrieval demo (query → chunks)
- Interactive CLI tool (src/query.py)
- Single query mode
- Batch query support
Metadata-aware filtering
- Chapter
- Section
- Pedagogical role
- Minimum similarity threshold

Implementation Details

Vector Store (src/vector_store.py):

Abstract VectorStore interface for pluggable backends
InMemoryVectorStore implementation
- Fast cosine similarity search with numpy
- Metadata filtering support
- Persistent storage (save/load)
- Statistics and analytics
build_vector_store_from_chunks() helper function

RAG Retriever (src/rag_retriever.py):

RAGRetriever class for semantic search
Convenience methods for role-based queries:
- get_definitions()
- get_examples()
- get_procedures()
- get_explanations()
multi_role_query() for querying multiple roles at once
Result formatting utilities

Query CLI (src/query.py):

Interactive query mode with REPL
Single query mode for scripting
Role and section filtering via command line
JSON output support
Built-in help and statistics

Pipeline Integration (src/pipeline.py):

--build-index flag to generate vector store
Automatic integration with semantic chunking
Reuses embedding cache from semantic chunking

Testing (tests/test_phase_5.py):

20+ unit tests covering:
- Vector store operations
- RAG retriever functionality
- Metadata filtering
- Save/load persistence
- End-to-end workflow

Usage

# Build index during processing
python src/main.py input.pdf --refine --semantic-chunk --classify-roles --build-index

# Query the index
python src/query.py data/output/book/vector_store.pkl --interactive

# Single query with role filter
python src/query.py data/output/book/vector_store.pkl \
  --query "What is a triangle?" --role definition --top-k 3

# Save results to JSON
python src/query.py data/output/book/vector_store.pkl \
  --query "calculate area" --role procedure -o results.json

Architecture

Local-first: No external APIs required
Fast: In-memory cosine similarity search
Portable: Pure Python with numpy
Extensible: Abstract interface for custom vector stores
Cached: Reuses embeddings from semantic chunking phase

✅ Phase 5.x — Persistence & Performance (DONE)

Goal: Upgrade storage and intelligent retrieval with LanceDB.

Implementation Complete

LanceDB Migration - Modern vector database with disk persistence
- PyArrow schema with 20+ metadata fields for SQL queries
- IVF-PQ indexing for 10-100x faster search on large collections
- Backward compatible with pickle fallback
- Module: src/vector_store_lancedb.py (618 lines)
SQL Filtering - Advanced query capabilities
- WHERE clauses with range queries (e.g., page_start >= 5 AND page_start <= 10)
- OR logic (e.g., role = 'definition' OR role = 'example')
- Numeric comparisons (e.g., role_confidence > 0.8)
- Integrated into RAGRetriever with query_sql() method
Migration Tool - Convert existing indexes
- Single file migration: .pkl → .lancedb
- Batch migration for directory trees
- Data integrity verification
- Module: src/migrate_to_lancedb.py (326 lines)
Pipeline Integration
- LanceDB by default with graceful fallback to pickle
- Automatic ANN indexing for datasets ≥256 chunks
- --build-index flag updated
Testing - Comprehensive test suite
- 18 unit tests in tests/test_lancedb_migration.py
- Coverage: CRUD, indexing, SQL filtering, migration, performance

Performance Metrics

Search latency: <50ms for 10k chunks (vs. 500ms with pickle)
Storage: ~500 KB per 40 chunks (LanceDB + embeddings)
Memory: <20 MB for 400 chunks (disk-backed streaming)

Usage

# Build LanceDB index
python src/main.py input.pdf --semantic-chunk --build-index

# Query with SQL filtering
python src/query.py vector_store.lancedb --query "triangles" \
  --sql-filter "page_start >= 5 AND page_start <= 10"

# Migrate existing pickle stores
python src/migrate_to_lancedb.py old.pkl new.lancedb
python src/migrate_to_lancedb.py --batch data/output data/output_lancedb

Deferred Features

Pedagogical bundling: auto-retrieve related definitions/procedures
Math consistency check: verify refinery quality

✅ Phase 5.v — Vision-RAG Integration (DONE)

Goal: Enable visual content retrieval alongside text-based RAG.

Implementation Complete

Visual ROI Extraction - Extract figures/tables from pages
- Crop with configurable padding (default: 10px)
- Auto-detect from Phase 4.x metadata (tables, figures, diagrams)
- Save as separate PNG files in rois/ directory
- Module: src/vision_extractor.py (360 lines)
CLIP Vision Embeddings - 512-dim image embeddings
- OpenAI CLIP ViT-B/32 integration
- Disk caching system (hash-based like text embeddings)
- Batch processing with GPU/CPU auto-detection
- Cross-modal search: text→image queries via embed_text_query()
- Module: src/vision_embeddings.py (390 lines)
ColPali Encoder (Optional) - Document-specialized vision
- CLIP fallback when ColPali unavailable
- Designed for tables and multi-column layouts
- Factory function for best encoder selection
- Module: src/colpali_encoder.py (220 lines)
Visual Chunk Linking - Connect ROIs to semantic chunks
- Page-based spatial proximity linking
- Visual metadata added to chunks
- Statistics and filtering utilities
- Module: src/visual_chunk_linker.py (180 lines)
LanceDB Schema Extension - Vision fields in vector store
- image_embedding (512-dim CLIP vectors)
- has_visual_content, image_path, visual_type, bbox fields
- add_vision_embeddings() method
- search_hybrid() with configurable text/vision weights
Hybrid Search - Text + vision retrieval
- query_hybrid() in RAGRetriever
- Configurable weights (default: 70% text, 30% vision)
- Text-only, vision-only, or combined modes
- Graceful fallback to text-only search
Pipeline Integration - End-to-end vision extraction
- New --extract-visuals CLI flag
- Extracts ROIs during pipeline execution
- Links to semantic chunks automatically
- Visual statistics in logs

Architecture: Dual-Stream Indexing

Stream A (Text): 768-dim sentence-transformers embeddings
Stream B (Vision): 512-dim CLIP image embeddings
Hybrid Search: Weighted combination with learned fusion

Usage

# Full vision-RAG pipeline
python src/main.py textbook.pdf \
  --layout-aware \
  --detect-tables \
  --detect-figures \
  --semantic-chunk \
  --extract-visuals \
  --build-index

# Hybrid text + image query (Python API)
from src.rag_retriever import load_retriever
from pathlib import Path

retriever = load_retriever(Path("vector_store.lancedb"))
results = retriever.query_hybrid(
    query_text="triangle area formula",
    query_image_path=Path("diagram.png"),
    text_weight=0.6,
    vision_weight=0.4
)

Output Structure

data/output/textbook/
├── images/              # Page images
├── rois/                # Extracted figures/tables (NEW!)
│   ├── roi_page0000_block0003_figure.png
│   ├── roi_page0002_block0012_table.png
│   └── ...
├── chunks_semantic.json # With visual metadata (NEW!)
└── vector_store.lancedb/ # LanceDB index (NEW!)

Deferred Features

Full vision embedding scoring in hybrid search (framework ready)
Qwen2-VL integration for visual reasoning
Vision-RAG test suite (1 remaining task)

🟢 Phase 6 — UX, DX & Stability (PARTIAL)

🟢 Phase 7 — The "Teacher's Assistant" Export

Goal: Make the data useful outside of the TextStruct CLI.

Knowledge Graph Export: Export JSON as a .dot or Mermaid file showing concept dependencies.
Anki/Markdown Export: Auto-generate study decks based on question and definition roles.

🎯 Current Recommended Focus

With Phases 3.y, 3.z, 3.zx, 4, 4.x, and 5 complete, the recommended next steps are:

Phase 2.x — Section Refinement (improve section accuracy in noisy textbooks)
Phase 3.z1 — Spatial-Semantic Linking (link text chunks to nearby visuals using Phase 4.x data)
Phase 5.v — Vision-RAG Integration (ColPali for complex math/layouts)
Phase 6 — UX/DX Improvements (logging, config management, performance stats)

🧠 Design Principles (LOCKED)

Local-first
CPU-safe by default
GPU is an optimization, never a requirement
Deterministic output
Inspectable intermediate artifacts
Model-agnostic where possible

FilesExpand file tree

ToDo.md

Latest commit

History

ToDo.md

File metadata and controls

🧱 TextStruct — Project ToDo & Roadmap

✅ Phase 0 — Foundation (DONE)

✅ Phase 1 — Text Quality & Cleanup (HIGH PRIORITY, DONE)

✅ Phase 2 — Section & Subsection Detection (VERY HIGH ROI, DONE)

🟡 Phase 2.x — Section Refinement (DEFERRED)

✅ Phase 3 — Chunk Builder (RAG-Ready Output, DONE)

Chunk Schema

Implementation

✅ Phase 3.x — Chunk Refinement (DONE)

🟢 Phase 3.y — Pedagogical Role Mapping (READY TO IMPLEMENT)

Architecture

Pedagogical Roles (Locked Vocabulary)

Implementation Breakdown

Phase 1 — Role Reference Library

Phase 2 — Heuristic Matching

Phase 3 — Main Classifier

Phase 4 — LLM Fallback

Phase 5 — Pipeline Integration

Phase 6 — CLI Validation

Phase 7 — Testing

Output Schema

✅ Phase 3.z3 — Metadata Refinement (DONE)

✅ Phase 3.zx — Math & OCR Cleanup (DONE)

Core Module Implementation

Angle Symbol Normalization

Q./Ans. Dialogue Labeling

Word Break Repair

Math Operator Spacing

Optional Ollama-Powered Math Fix

Testing & Integration

Output Schema

Deferred Features

✅ Phase 4 — Layout-Aware Processing (DONE)

Implementation Complete

Architecture

CLI Usage

✅ Phase 4.x — Advanced Layout Features (DONE)

Implementation Complete

Data Model Extensions

Pipeline Integration

CLI Flags

Testing

Deferred Features

✅ Phase 5 — Embeddings & RAG Integration (DONE)

Implementation Details

Usage

Architecture

✅ Phase 5.x — Persistence & Performance (DONE)

Implementation Complete

Performance Metrics

Usage

Deferred Features

✅ Phase 5.v — Vision-RAG Integration (DONE)

Implementation Complete

Architecture: Dual-Stream Indexing

Usage

Output Structure

Deferred Features

🟢 Phase 6 — UX, DX & Stability (PARTIAL)

🟢 Phase 7 — The "Teacher's Assistant" Export

🎯 Current Recommended Focus

🧠 Design Principles (LOCKED)