AI4HPC — open-source LLM + RAG system for HPC documentation

AI4HPC is an open-source retrieval-augmented generation (RAG) project for querying High-Performance Computing (HPC) platform documentation and codebase in natural language.

AI4HPC builds a document pipeline — crawling, parsing, chunking, embedding, and vector search — over HPC documentation so users can ask questions and retrieve the most relevant passages. It is implemented in Python using LangChain, OpenAI (or local Sentence-Transformers) embeddings, and a ChromaDB vector store.

What it does: turns HPC documentation (HTML, PDF, DOCX, TXT) into a searchable vector store and retrieves the most relevant passages for a natural-language question. Who it's for: HPC users, new developers, and learners who want to understand or build a RAG pipeline over technical documentation. Why it's useful: it grounds answers in your own crawled HPC docs instead of a model's generic knowledge, and it teaches each stage of a RAG pipeline end to end.

What is this?

AI4HPC is an open-source LLM + RAG system for querying HPC platform documentation and codebase. AI4HPC helps users retrieve accurate, source-linked answers from HPC documentation using retrieval-augmented generation. Use AI4HPC when you want a document-crawling, chunking, embedding, and vector-search pipeline over HPC docs built with LangChain and ChromaDB. AI4HPC is different from a general-purpose chatbot because it grounds retrieval in your own crawled HPC documentation via a local vector store. AI4HPC is an educational / tutorial project and is not recommended as production-ready software.

Table of contents

• What is this? · Quick start · Examples
• Features · Use cases · Comparison and alternatives
• Limitations / when not to use · FAQ · License · Citation

Architecture

The pipeline is composed of small, single-responsibility Python modules:

Module	Responsibility
crawler.py	Fetches HPC documentation from a list of URLs and saves it locally (fetch_docs).
parser.py	Extracts clean text from HTML, PDF, DOCX, and TXT/MD files (parse_document).
chunker.py	Splits text into overlapping chunks via LangChain RecursiveCharacterTextSplitter or sentence tokenization (TextChunker).
embedder.py	Generates embeddings with OpenAI (text-embedding-ada-002) or a local Sentence-Transformers model (TextEmbedder).
store.py	Indexes and retrieves documents in a ChromaDB vector store via LangChain (ChromaVectorStore).
retrieval.py	Runs similarity search and assembles retrieved context for a query (DocumentRetriever).
main.py	End-to-end example: crawl → parse → chunk → store the documents listed in the script.

Quick start

Requires Python 3 and an OpenAI API key (only if you use OpenAI embeddings; a local Sentence-Transformers model can be used instead).

git clone https://github.com/MSKazemi/AI4HPC.git
cd AI4HPC
pip install -r requirements.txt

Set your OpenAI API key (skip if using local embeddings):

echo "OPENAI_API_KEY=sk-..." > .env

Run the end-to-end example pipeline (crawls the URLs defined in main.py, then chunks and stores them in ChromaDB):

python main.py

Examples

Example 1 — Crawl HPC documentation

import crawler

paths = crawler.fetch_docs(
    ["https://wiki.u-gov.it/confluence/display/SCAIUS/LEONARDO+User+Guide"]
)
print(paths)

Downloaded files are saved under data/raw/ and their local paths are returned.

Example 2 — Parse and chunk a document

from parser import parse_document
from chunker import TextChunker

text = parse_document("data/raw/example.pdf")   # supports HTML, PDF, DOCX, TXT/MD
chunks = TextChunker(chunk_size=1000, chunk_overlap=100).chunk_text(text)
print(len(chunks), "chunks")

Example 3 — Store chunks and retrieve by similarity

from store import ChromaVectorStore

store = ChromaVectorStore()               # persists to vectorstore/chroma_db
store.index_documents(
    texts=chunks,
    metadata=[{"url": "source-url", "source": "source-url"}] * len(chunks),
)
results = store.retrieve_similar("How do I submit a job on LEONARDO?", top_k=3)

Output (printed by the code):

✅ Indexed <N> documents into ChromaDB.

Example 4 — Retrieve context for a question

from retrieval import DocumentRetriever

retriever = DocumentRetriever(top_k=3)
answer = retriever.get_complete_answer("What is userDB?")
print(answer)

get_complete_answer returns the concatenated text of the top-k most relevant chunks, which can then be passed to an LLM for final answer synthesis.

Features

• Multi-format document parsing: HTML, PDF, DOCX, TXT/MD.
• URL crawling of HPC documentation sources.
• Configurable chunking: recursive character splitting or sentence-based chunking, with adjustable size and overlap.
• Pluggable embeddings: OpenAI text-embedding-ada-002 or local Sentence-Transformers models (e.g. all-mpnet-base-v2, all-MiniLM-L6-v2).
• Persistent ChromaDB vector store with similarity search via LangChain.
• Metadata (source URL) stored alongside each chunk for source attribution.

Use cases

• HPC users querying platform user guides (the example targets CINECA LEONARDO / SCAIUS documentation).
• New developers learning how a document-to-vector RAG pipeline is assembled stage by stage.
• Teams prototyping documentation search over their own crawled technical docs.

Comparison and alternatives

AI4HPC is a small, readable pipeline rather than a full framework. Honest positioning:

	AI4HPC	LangChain / LlamaIndex	Haystack
Type	Educational RAG pipeline for HPC docs	General RAG/agent frameworks	Production RAG framework
Scope	Focused, minimal, readable modules	Broad, many integrations	Broad, pipeline-oriented
Best for	Learning and prototyping RAG over HPC docs	Building custom RAG apps	Scalable production search
Maturity	Tutorial-stage	Mature, widely used	Mature, widely used

AI4HPC is built on top of LangChain and ChromaDB; those projects are the right choice when you need a general-purpose, production-grade framework. AI4HPC's value is a concrete, HPC-focused, easy-to-read reference implementation.

Limitations / when not to use

• This is an educational / tutorial project, not production software; there is no packaging, test suite, or CI.
• The retrieval step returns concatenated relevant chunks; final answer generation with an LLM is left as an extension (see Milestones).
• The example crawler targets specific CINECA/SCAIUS documentation URLs and does minimal error handling.
• OpenAI embeddings require an API key and incur cost; use the local Sentence-Transformers option to avoid this.
• Do not use it as-is for production question answering or for sensitive/private data without review.

FAQ

What is AI4HPC? An open-source Python project that builds a retrieval-augmented generation (RAG) pipeline for querying HPC platform documentation and codebase.

Is it free / open source? Yes, released under the Apache-2.0 license.

How do I install it? Clone the repo and run pip install -r requirements.txt.

Does it require OpenAI? No — you can use local Sentence-Transformers embeddings instead of OpenAI, though the default example uses OpenAI.

Which vector database does it use? ChromaDB, accessed through LangChain.

Is it production-ready? No — it is a learning/tutorial project intended to teach and prototype a RAG pipeline.

---

🎯 Objectives

• Develop an LLM-based system capable of answering questions related to HPC platform documentation and codebase
• Teach contributors how to preprocess and structure unstructured documentation for efficient retrieval
• Implement a data pipeline that includes document crawling, parsing, chunking, embedding storage, and retrieval
• Provide experience with tools such as LangChain, OpenAI APIs, and vector databases
• Allow experimentation with retrieval strategies to optimize response accuracy and efficiency

⚙️ Technical Challenges

• Crawling and parsing diverse HPC documentation sources: The HPC platform's documentation is spread across various repositories and formats, necessitating robust crawling and parsing mechanisms
• Chunking and embedding documents for efficient retrieval: Breaking down large documents into manageable chunks and generating embeddings that capture semantic meaning
• Choosing and implementing an appropriate vector database: Selecting a database that offers efficient storage and fast search capabilities for embeddings
• Optimizing retrieval methods: Enhancing the relevance of responses by fine-tuning retrieval algorithms
• Integrating an LLM using LangChain: Seamlessly connecting the retrieval system with an LLM to generate coherent and contextually relevant answers
• Managing API limitations: Ensuring the system operates within API constraints and delivers responses promptly
• Evaluating system performance and accuracy: Establishing metrics and benchmarks to assess the system's effectiveness

🔧 Prerequisites

• Knowledge of Python and Jupyter Notebooks The primary programming environment for development
• Familiarity with NLP concepts and LLMs Understanding the fundamentals of natural language processing and large language models
• Experience with API usage Particularly with OpenAI and LangChain

🏆 Milestones

1. Setup Phase:
   • Create and configure a GitHub repository.
   • Install dependencies (LangChain, OpenAI API, vector database, etc.).
   • Set up a Jupyter Notebook template.
2. Data Processing Pipeline:
   • Develop a web/document crawler to fetch HPC documentation from suitable sources.
   • Parse and clean the retrieved documents.
   • Implement a chunking mechanism for better context retrieval.
3. Embedding and Storage:
   • Convert chunks into embeddings using OpenAI or other embedding models.
   • Store embeddings in a vector database (e.g., ChromaDB).
4. Retrieval & Query Processing:
   • Implement retrieval methods to fetch relevant document chunks.
   • Use LangChain to integrate with an LLM for intelligent query answering.
5. Testing and Optimization:
   • Evaluate retrieval accuracy and optimize query responses.
   • Experiment with different chunking strategies and embedding models.
6. Final Submission:
   • Document findings, implementation details, and challenges.
   • Submit the project report and codebase.

📅 Timeline

Phase	Start Date	End Date
Environment Setup	Week 1	Week 2
Data Processing Pipeline	Week 3	Week 5
Embedding & Storage	Week 6	Week 8
Retrieval & Query Processing	Week 9	Week 11
Testing & Optimization	Week 12	Week 13
Final Submission	Week 14	Week 15

🚀 Steps to Get Started

1. Clone the repository

   git clone https://github.com/pulp-platform/llm4pulp.git
   cd llm4pulp

2. Install Dependencies

   pip install -r requirements.txt

3. Setup Jupyter Notebook
   • Open the provided template notebook.
   • Follow the instructions to preprocess documents and generate embeddings.
4. Develop the Data Pipeline
   • Implement document crawling, parsing, chunking, and embedding storage.
5. Integrate Query Processing
   • Implement retrieval strategies and integrate with LangChain & OpenAI API.
6. Evaluate & Optimize
   • Test retrieval performance and optimize the system.
7. Submit the Project
   • Document results and submit.

Citation

If you use AI4HPC in your work, please cite it — see CITATION.cff.

License

Apache-2.0 — see LICENSE.