Cloud-Powered RAG Chatbot

A Python application that answers questions about your local documents using Retrieval-Augmented Generation (RAG).

The application reads text, JSON, and CSV files, splits them into chunks, embeds those chunks through an OpenAI-compatible cloud API, and stores their vectors locally in FAISS. When you ask a question, it retrieves matching chunks and sends them to a cloud language model to generate an answer.

Both the embedding model and the language model support configurable model names, API keys, and base URLs.

Features

• Load documents from .txt, .json, and .csv files.
• Split documents into overlapping chunks.
• Generate cloud embeddings using the OpenAI Python SDK.
• Store and search vectors locally using FAISS.
• Generate answers using retrieved document context.
• Display the sources used for each answer.
• Configure separate endpoints for embeddings and chat.
• Check that the configured embedding model matches the model used to build the index.
• Configure settings through environment variables or a .env file.

How It Works

Ingestion

Local documents
      │
      ▼
Load document content
      │
      ▼
Split into overlapping chunks
      │
      ▼
Generate cloud embeddings
      │
      ▼
Normalize vectors
      │
      ▼
Save vectors and chunk metadata locally

Chat

User question
      │
      ▼
Generate a question embedding
      │
      ▼
Retrieve matching chunks from FAISS
      │
      ▼
Send question and retrieved context to the LLM
      │
      ▼
Display answer and sources

The generator is instructed to answer using only the retrieved context and to acknowledge when the available data does not contain an answer. This is a prompting instruction, not a guarantee that every generated answer will be correct.

Requirements

• Python 3.10 or newer
• An embedding API compatible with the OpenAI SDK's embeddings interface
• A chat API compatible with the OpenAI SDK's chat completions interface
• API credentials for the selected services
• An internet connection for embedding and generation requests

The application does not require a local embedding model or a GPU.

Project Structure

Key files and directories:

.
├── main.py                 # Application entry point
├── config.py               # Environment-based configuration
├── requirements.txt        # Python dependencies
├── .env.example            # Example environment configuration
├── data/                   # Input documents
├── index_store/            # Saved FAISS index and metadata
└── rag/
    ├── loaders.py          # Document loading
    ├── chunker.py          # Document chunking
    ├── embedder.py         # Cloud embedding requests
    ├── vector_store.py     # FAISS storage and persistence
    ├── ingest.py           # Ingestion pipeline
    ├── retriever.py        # Similarity retrieval
    ├── generator.py        # Context-based answer generation
    └── chat.py             # Interactive chat loop

Installation

Run the following commands from the project directory.

1. Create a virtual environment

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Activate it in Windows PowerShell:

.venv\Scripts\Activate.ps1

2. Install dependencies

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

The application uses:

Package	Purpose
faiss-cpu	Local vector indexing and similarity search
numpy	Vector processing and normalization
openai	Cloud embedding and chat API requests
python-dotenv	Loading configuration from .env

3. Create your environment file

On macOS or Linux:

cp .env.example .env

On Windows PowerShell:

Copy-Item .env.example .env

Edit .env and add your API credentials.

Configuration

Use the Same Provider for Embeddings and Chat

OPENAI_API_KEY=your_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1

LLM_MODEL=gpt-4o-mini
EMBEDDING_MODEL=text-embedding-3-small

When embedding-specific credentials are not set, the embedder uses OPENAI_API_KEY and OPENAI_BASE_URL.

Use Separate Providers

# Chat generation
OPENAI_API_KEY=your_chat_api_key
OPENAI_BASE_URL=https://your-chat-provider.example/v1
LLM_MODEL=your-chat-model

# Embeddings
EMBEDDING_API_KEY=your_embedding_api_key
EMBEDDING_BASE_URL=https://your-embedding-provider.example/v1
EMBEDDING_MODEL=your-embedding-model

Replace the example URLs and model names with values supported by your providers.

A custom base URL must expose the appropriate OpenAI-compatible API routes. Changing the base URL alone does not make an incompatible service work.

Available Settings

Variable	Default	Description
DATA_DIR	<project>/data	Directory containing input documents
INDEX_DIR	<project>/index_store	Directory for the saved index and metadata
CHUNK_SIZE	800	Chunk size in characters
CHUNK_OVERLAP	100	Overlap between chunks in characters
OPENAI_API_KEY	Empty	API key for chat generation
OPENAI_BASE_URL	https://api.openai.com/v1	Chat API base URL
LLM_MODEL	gpt-4o-mini	Chat model name
LLM_MAX_TOKENS	1024	Output token limit sent to the chat API
EMBEDDING_MODEL	text-embedding-3-small	Embedding model name
EMBEDDING_API_KEY	Value of OPENAI_API_KEY	Embedding API key
EMBEDDING_BASE_URL	Value of OPENAI_BASE_URL	Embedding API base URL
EMBEDDING_BATCH_SIZE	64	Number of texts per embedding request
TOP_K	4	Number of chunks requested during retrieval

Use a positive embedding batch size and a chunk overlap smaller than the chunk size.

To use the credential fallback, leave the embedding-specific variables unset rather than setting them to empty strings.

Usage

1. Add Your Documents

Create the input directory if necessary:

mkdir -p data

On Windows PowerShell:

New-Item -ItemType Directory -Force data

Add supported files:

data/
├── product_guide.txt
├── policies.json
└── inventory.csv

Supported extensions are:

• .txt
• .json
• .csv

Other formats, such as PDF and Word documents, are not supported by the current configuration.

2. Build the Index

python main.py ingest

The ingestion pipeline:

1. Reads documents from DATA_DIR.
2. Splits them into overlapping chunks.
3. Sends the chunk text to the embedding API in batches.
4. Converts the returned vectors to normalized float32 arrays.
5. Builds a FAISS index.
6. Saves the index, chunk metadata, and embedding model name to INDEX_DIR.

The first ingestion must finish successfully before you start chatting.

3. Start the Chat

python main.py chat

Example interaction:

Loaded 24 chunk(s). Ask a question (type 'exit' to quit).

You > What is the return policy?

Assistant > According to the provided policy, ...

Sources:
  - policies.json (score: 0.812)

Source scores represent vector similarity, not calibrated confidence in the generated answer.

To exit, enter:

exit

You can also enter quit or q, or press Ctrl+C.

4. Rebuild When Needed

Run ingestion again after:

• Adding, removing, or editing documents
• Changing chunk size or overlap
• Changing the embedding model
• Switching to an endpoint that serves a different embedding model, even if its model name is unchanged

python main.py ingest

The model-name check helps prevent querying an index with a different configured model. It does not detect changes hidden behind the same model name.

Changing only the chat model does not require rebuilding the index.

Implementation Notes

Cloud Embeddings

rag/embedder.py calls:

client.embeddings.create(
    model=self.model_name,
    input=batch,
)

Returned embeddings are sorted by their response index to preserve input order.

Vectors are L2-normalized before storage and retrieval so that inner-product similarity corresponds to cosine similarity for nonzero vectors.

The embedder discovers the vector dimension from the returned embeddings. If its dimension property is accessed before any text has been embedded, it sends a small probe request.

Answer Generation

rag/generator.py sends the retrieved context and current question through:

client.chat.completions.create(...)

The current chat loop does not send previous turns to the language model. Each question is answered independently using its retrieved context.

Model Compatibility

The generator currently uses max_tokens.

If your chosen model rejects that parameter and requires max_completion_tokens, update the generation call in rag/generator.py accordingly.

Other provider-specific requirements may also need code changes.

Privacy, Security, and Costs

• Document chunks are sent to the configured embedding provider.
• Each chat question is sent to the embedding provider.
• The question and retrieved chunks are sent to the chat provider.
• The FAISS index and chunk metadata remain in the configured local index directory.
• Ingestion and chat requests may incur API charges.
• Provider data handling and retention policies apply to content sent to their services.

Only ingest documents that you are authorized to process with the selected providers.

Do not commit API credentials or sensitive document data. Consider adding these entries to .gitignore:

.env
.venv/
__pycache__/
*.pyc

# Local documents and generated index
data/
index_store/

Retrieved documents are external input. The current prompt is not a robust defense against malicious instructions embedded in documents.

Troubleshooting

Missing API Key

Set the relevant key in .env:

OPENAI_API_KEY=your_chat_api_key
EMBEDDING_API_KEY=your_embedding_api_key

If both services use the same key, leave EMBEDDING_API_KEY unset.

Index Not Found

Build the index first:

python main.py ingest

Confirm that ingestion and chat use the same INDEX_DIR.

Embedding Model Mismatch

Rebuild the index with the configured model, or restore the model that originally created it:

python main.py ingest

Authentication Errors or Model Not Found

Check that:

• The API key is valid.
• The base URL is correct.
• The model name is available through that endpoint.
• The provider supports the requested API interface.

Embedding and chat endpoints can have different supported models.

Rate Limits or Oversized Requests

Try reducing:

EMBEDDING_BATCH_SIZE=16

For input-length errors, consider reducing CHUNK_SIZE and rebuilding the index. Chunk sizes are measured in characters, while providers typically enforce token limits.

Poor or Unsupported Answers

Check that the source documents contain the answer and that the index is current.

You can also experiment with:

TOP_K=6
CHUNK_SIZE=1000
CHUNK_OVERLAP=150

Rebuild the index after changing chunking settings. Changing TOP_K does not require rebuilding.

Larger retrieval counts can increase chat request size and cost, and may introduce less relevant context.

Current Limitations

• Supports text, JSON, and CSV documents only.
• Requires cloud API access for ingestion and answering.
• Does not maintain conversational history.
• Does not automatically detect document changes.
• Checks embedding compatibility by model name, not provider identity or model version.
• Displays retrieved source names but does not enforce inline citations.
• Cannot guarantee factual accuracy or that every answer is supported by the retrieved context.

---

Generated by: GPT-6.1 Sol