🔬 PaperLens

Semantic research paper discovery — search arXiv, cluster by topic, and summarize key findings using NLP and unsupervised learning.

---

🧠 What It Does

PaperLens takes a research query (or a PDF paper you're reading) and automatically:

1. Fetches the most relevant papers from arXiv — always up to date, no static dataset
2. Embeds abstracts using sentence-transformers (all-MiniLM-L6-v2)
3. Clusters papers by topic using UMAP dimensionality reduction + HDBSCAN
4. Summarizes each cluster's key findings using a local LLM via Ollama
5. Visualizes the semantic landscape as an interactive 2D scatter plot

---

🖥️ Demo

---

⚙️ Architecture

User Query / PDF Upload
        │
        ▼
  arXiv API (live fetch)
        │
        ▼
  sentence-transformers
  (all-MiniLM-L6-v2 embeddings)
        │
        ▼
  UMAP (384D → 10D)
        │
        ▼
  HDBSCAN Clustering
        │
        ├──► UMAP (10D → 2D) → Plotly Scatter Map
        │
        └──► Ollama (LLaMA 3.2) → Cluster Summaries
        │
        ▼
  Gradio UI

---

🚧 Current Limitations & Roadmap

PaperLens is currently built as a lightweight, on-the-fly prototype. Because it relies on a live keyword search via the arXiv API for its initial paper retrieval, the downstream clustering and summarization are heavily bottlenecked by the relevance of those initial keyword search results.

To evolve into a production-scale discovery tool with excellent relevance, the architecture would need to transition to a Vector Database model:

1. Pre-computed Database: An ingestion pipeline that fetches arXiv daily dumps, embeds abstracts using an academic-focused model (like allenai/specter2_base), and stores them in a Vector DB.
2. True Semantic Search: Instead of relying on arXiv's keyword API, the app would embed the user's query and perform an instant Nearest Neighbor search across the entire database.

This upgrade would eliminate rate limits, drop retrieval times to milliseconds, and guarantee that only the most conceptually relevant papers are clustered.

---

🚀 Setup

1. Clone the repo

git clone [https://github.com/yourusername/paperlens.git](https://github.com/yourusername/paperlens.git)
cd paperlens

2. Create conda environment

conda create -n paperlens python=3.11 -y
conda activate paperlens
pip install -r requirements.txt

3. Install and start Ollama

Download Ollama from ollama.com, then:

ollama pull llama3.2
ollama serve

4. Run the app

python app.py

Open http://localhost:7860 in your browser.

---

📦 Tech Stack

Component	Library
UI	Gradio
Embeddings	sentence-transformers (all-MiniLM-L6-v2)
Similarity Search	FAISS
Dimensionality Reduction	UMAP
Clustering	HDBSCAN
Summarization	Ollama (LLaMA 3.2)
PDF Parsing	PyMuPDF
Visualization	Plotly
Paper Source	arXiv API

---

🔍 Features

• Live arXiv search — results are always current, no database needed
• Semantic clustering — groups papers by meaning, not just keywords
• LLM summarization — key findings per cluster, powered by a local model
• Interactive cluster map — hover over any dot to see paper title and authors
• PDF upload — upload a paper you're reading and auto-extract a search query
• Fully local — no external API keys required, runs entirely on your machine

---

📁 Project Structure

paperlens/
├── app.py                  # Gradio UI and pipeline orchestration
├── src/
│   ├── fetch.py            # arXiv API integration
│   ├── embed.py            # Sentence embedding
│   ├── cluster.py          # UMAP + HDBSCAN clustering
│   ├── summarize.py        # Ollama LLM summarization
│   ├── visualize.py        # Plotly scatter plot
│   └── pdf_extract.py      # PDF query extraction
├── requirements.txt
└── README.md

---

💡 Example Queries

• transformer attention mechanism
• diffusion models image generation
• reinforcement learning robotics
• large language model reasoning
• federated learning privacy

---

📄 License

MIT License — free to use, modify, and distribute.