Add on-demand RAG knowledge base

.gitignore

@@ -153,4 +153,5 @@ cython_debug/
 # sqlite database
 *.db
 
-.env
+.env
+private.txt

CLAUDE.md

@@ -0,0 +1,94 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+A Python multi-agent orchestration system built on LangChain/LangGraph, exposed via three interfaces (REST API, CLI, MCP server) that all share the same agent/routing/DB layer. Observability is via Langfuse; models are accessed through LiteLLM (currently Gemini 2.5 Flash).
+
+## Commands
+
+```bash
+# Setup
+uv venv venv
+source venv/bin/activate
+touch main.db          # create empty sqlite db
+uv sync                # install deps from pyproject.toml
+
+# Run REST API (http://127.0.0.1:8000, docs at /docs)
+uvicorn app.main:app --reload
+
+# Run interactive CLI
+python -m cli
+
+# Run MCP server (stdio, for Claude Desktop / Cursor)
+./venv/bin/python mcp_server.py
+
+# Run MCP server (SSE/HTTP, http://localhost:8000/sse)
+python -m mcp_server sse
+
+# Run all tests
+PYTHONPATH=. pytest
+
+# Run a single test file / test
+PYTHONPATH=. pytest tests/test_weather_agent.py
+PYTHONPATH=. pytest tests/test_weather_agent.py::test_choose_agent_routing
+```
+
+Use `./venv/bin/python` / `./venv/bin/pytest` explicitly when the venv isn't activated (per `.agents/AGENTS.md`).
+
+Tests set required env vars (mock API keys, `DATABASE_URL=sqlite:///main_test.db`) in `tests/conftest.py`, and use an in-memory SQLite engine + `app.dependency_overrides` per test module — no `.env` needed to run the suite.
+
+## Architecture
+
+### Three entrypoints, one core
+
+`app/main.py` (FastAPI), `cli.py`, and `mcp_server.py` are thin wrappers around the same core: `app.agents.choose_agent` for routing + LangGraph agent graphs (`car_hire_agent`, `weather_agent`, `generate_image_agent`, `marketing_agent`, `rag_ingest_agent`, `rag_query_agent`) for execution. All three entrypoints duplicate the same per-agent "build initial state / invoke / handle `next_question`" dispatch logic — when changing an agent's state shape or adding an agent, update all three call sites plus `app/agents/__init__.py`.
+
+### Request flow (REST, mirrored by CLI/MCP)
+
+1. `POST /llm-job` (`app/routes/llm_job.py`) creates an `LLMJob` row (`app/models/llm_job.py`), status `queue`.
+2. `choose_agent()` (`app/agents/agent_chooser.py`) routes the prompt. Before calling the LLM it short-circuits through `app/agents/agent_memory.py`:
+   - `try_handle_profile` — handles "remember X" / "who am I" locally.
+   - `check_prompt_cache` — exact-match cache for prior `direct_answer`/`image` responses, and a 15-minute TTL cache for `weather`.
+   - Only if neither hits does it call Gemini with structured output to pick an action (`car_hire_agent` / `weather_agent` / `generate_image_agent` / `marketing_agent` / `rag_ingest_agent` / `rag_query_agent` / `direct_answer` / `unsupported`). Before this call it also fetches the most recent `RagDocument` URLs (`get_known_rag_sources`) and includes them in the router's system prompt, so questions about previously-ingested sites route to `rag_query_agent` instead of `direct_answer`/`unsupported`.
+3. Direct/unsupported answers finish the job immediately (status `done`). Otherwise the matching LangGraph agent is invoked with `config={"configurable": {"session": ..., "user_id": ...}}` so agent nodes can read/write `LLMMemory` (`app/models/llm_memory.py`) via the DB session.
+4. If the agent graph returns `next_question` (a field on its `TypedDict` state), the job goes to status `awaiting_input` and the response is the follow-up question; the client resumes it via `PATCH /llm-job/{id}` with `{"answer": ...}`, which re-invokes the same agent (`agent_name` read back from `job.state["agent"]`) with prior state fields merged in. Otherwise the job is `done` (or `error` if the agent raised `AppException`/an exception).
+
+### LangGraph agents (`app/agents/`)
+
+Each agent (`car_hire_agent.py`, `weather_agent.py`, `generate_image_agent.py`, `marketing_agent.py`) follows the same shape: a `TypedDict` state, Pydantic schemas for structured LLM output, node functions built as `ChatPromptTemplate | llm.with_structured_output(...)`, and a compiled `StateGraph` with conditional edges for the human-in-the-loop (`missing_fields`/`next_question`) or approval (`marketing_agent`'s `approved_option`) pattern. Every LLM-calling node passes `get_langfuse_handler()` as a callback and tags calls with `langfuse_tags: [agent_name, node_name]` — follow this pattern for new nodes/agents so traces stay attributable in Langfuse.
+
+`rag_ingest_agent.py`/`rag_query_agent.py` are a linear (no HITL) pair implementing the URL-RAG flow from `specs/url-rag.md`: `rag_ingest_agent` regex-extracts a URL from the prompt (no LLM call — deterministic and cheaper than structured extraction for this), skips re-scraping if the URL is already in `RagDocument`, otherwise scrapes with `httpx` + `beautifulsoup4`, splits with `langchain_text_splitters.RecursiveCharacterTextSplitter`, and embeds/persists chunks via `GoogleGenerativeAIEmbeddings` (`app.config.GEMINI_EMBEDDING_MODEL`, reuses `GEMINI_API_KEY`). `rag_query_agent` embeds the question, ranks all `RagChunk` rows by cosine similarity (plain Python, no vector store — retrieval is brute-force, fine at this scale but not built to scale past a small KB), and synthesizes an answer from the top matches, falling back to a fixed "don't know" response below the similarity threshold rather than hallucinating.
+
+`agent_chooser.py` is the router; new agents must be registered in its system prompt (action name + routing description), re-exported from `app/agents/__init__.py`, and wired into the dispatch blocks in `app/main.py`/`routes/llm_job.py`, `cli.py`, and `mcp_server.py`.
+
+### Persistence
+
+- `app/db.py`: single SQLModel `engine` (SQLite by default), `init_db()` creates tables and seeds two demo `LLMJob` rows if empty, `get_session()` is the FastAPI dependency.
+- `LLMJob` (`app/models/llm_job.py`): the job/task record — `prompt`, `response`, `status` (`queue`/`awaiting_input`/`done`/`error`), `state` (JSON blob holding the active agent name + its in-progress fields for resumption).
+- `LLMMemory` (`app/models/llm_memory.py`): generic key/value cache keyed by `(user_id, memory_type, query_key)` where `query_key` is normalized via `normalize_key()` (lowercased, trimmed, trailing punctuation stripped). `memory_type` values in use: `profile`, `direct_answer`, `image`, `weather`, `weather_data`.
+- `RagDocument`/`RagChunk` (`app/models/rag.py`): the URL-RAG knowledge base — one `RagDocument` per ingested URL, one `RagChunk` per text chunk with its embedding stored as JSON (`Column(JSON)`, same pattern as `LLMJob.state`). Unlike `LLMMemory`, this isn't per-`user_id` — all ingested content is shared/global across users.
+
+### Model/config layer
+
+- `app/config.py` loads all env vars (via `.env`) as module-level constants — always add new secrets/settings here rather than reading `os.environ` elsewhere.
+- `app/llm_model.py` centralizes model construction (currently only `get_gemini_2_5_flash_model`, using `ChatLiteLLM`); `car_hire_agent.py` still constructs `ChatLiteLLM` directly instead of using this helper (inconsistency to be aware of, not necessarily to copy).
+- `app/langfuse.py` lazily builds and caches a single `CallbackHandler`; `get_langfuse_handler()` returns `None` if keys aren't configured, and call sites are expected to handle that (`config = {...} if handler else {}`).
+- `app/util.py` provides a single process-lifetime `get_session_id()` used as the Langfuse session id.
+
+## Repository Conventions
+
+From `.agents/AGENTS.md` (binding for all coding agents in this repo):
+
+- **Style**: PEP 8, `snake_case` functions/vars, `PascalCase` classes, `UPPER_SNAKE_CASE` constants, 4-space indents, two blank lines around top-level defs/classes, one blank line between methods.
+- **Imports**: absolute imports rooted at the project (`from app.config import X`, never bare `from config import X`); expose package members via `__init__.py` `__all__` where applicable (see `app/models/__init__.py`).
+- **SQLite**: never use the `COMMENT` keyword in SQL; always manage DB sessions via context managers or FastAPI `Depends`.
+- **FastAPI/Pydantic**: use `lifespan` (not `@app.on_event`), Pydantic v2's `.model_dump()` (not `.dict()`), `typing.Annotated` for route dependencies.
+- Read config from `app.config`, not `os.environ`, directly.
+- All prompts sent to an LLM must go through Langfuse observability (pass the callback handler + `langfuse_tags`/`langfuse_session_id` metadata, matching existing node patterns).
+- `TODO` comments mark code intentionally left alone — do not change unless explicitly asked.
+
+## Domain-specific skills
+
+`.agents/skills/` contains reference skills for LangChain/LangGraph/Langfuse/deep-agents patterns (fundamentals, RAG, middleware, persistence, human-in-the-loop, orchestration, etc.). Consult the relevant skill under this directory before making non-trivial changes to agent graphs, memory/persistence, or Langfuse instrumentation.

README.md

@@ -3,16 +3,28 @@
 A clean, python implementation to orchestrated multi Agentic AI with LangChain/LangGraph & Langfuse (observability).
 
 ## Features
-1. Multi Agents, system will chose which AI agent suitable for the tasks.
+1. Multi Agents, the system will choose which AI agent is suitable for the tasks.
 2. Memory capability.
-3. Model agnostic, easily switch between model that suit the task.
+3. Model agnostic, easily switch between models that suits the task.
 4. Multi platform: CLI, MCP Server, RESTful API.
+5. On-demand RAG knowledge base: add any URL and query its content afterward.
 
 ## Available Agents
 - `agent_chooser`: decide which agent specialized agent to use. If can answer directly (e.g: simple question, answer directly)
 - `agent_memory`: agent that remember the previous question and use it as base answer. Not to be confused with full RAG capability.
 - `weather_agent`: agent that can answer weather related question, powered by Open Weather API
 - `image_agent`: agent that can generate image.
+- `rag_ingest_agent`: scrapes a given URL, chunks and embeds its content, and stores it in the knowledge base (SQLite). Skips re-scraping if the URL was already added.
+- `rag_query_agent`: answers questions by retrieving the most relevant chunks from the knowledge base (via embedding similarity) and synthesizing an answer from them. Replies that it doesn't know if nothing relevant has been added yet.
+
+### RAG Knowledge Base Example
+```
+You: Add https://www.kayak.co.id/ to rag knowledge base
+Agent: Added 'https://www.kayak.co.id/' to the knowledge base (12 chunks indexed).
+
+You: What is kayak.co.id?
+Agent: Kayak.co.id is a travel search engine that compares prices across... (Source: https://www.kayak.co.id/)
+```
 
 ---
 
@@ -36,7 +48,7 @@ source venv/bin/activate
 touch main.db
 ```
 
-Create the `.env` file, see file [config.py](./app/config.py) for env used.
+Create the `.env` file, see the file [config.py](./app/config.py) for env used.
 
 ### 3. Install Dependencies
 ```bash
@@ -124,4 +136,6 @@ The project defines the following packages in `pyproject.toml`:
 2. **`uvicorn[standard]`**: An ASGI (Asynchronous Server Gateway Interface) web server implementation for Python. FastAPI is built on ASGI standard, and `uvicorn` acts as the server to run the FastAPI application. The `[standard]` extra installs high-performance loop dependencies like `uvloop` and `httptools`.
 3. **`pydantic`**: Data validation and settings management using Python type annotations. FastAPI uses Pydantic to parse and validate request JSON payloads, and serialize response objects.
 4. **`pytest`**: A robust testing framework for writing clean, readable, and scalable unit tests.
-5. **`httpx`**: A next-generation HTTP client for Python. It is used in unit tests alongside FastAPI's `TestClient` to make asynchronous/synchronous HTTP requests to the application.
+5. **`httpx`**: A next-generation HTTP client for Python. It is used in unit tests alongside FastAPI's `TestClient` to make asynchronous/synchronous HTTP requests to the application. Also used by `rag_ingest_agent` to fetch page content.
+6. **`beautifulsoup4`**: HTML parser used by `rag_ingest_agent` to strip tags and extract readable text from scraped pages.
+7. **`langchain-text-splitters`**: Splits scraped page text into overlapping chunks before embedding, for the RAG knowledge base.

app/agents/__init__.py

@@ -3,3 +3,5 @@
 from .agent_chooser import choose_agent
 from .generate_image_agent import generate_image_agent
 from .marketing_agent import marketing_agent
+from .rag_ingest_agent import rag_ingest_agent
+from .rag_query_agent import rag_query_agent

app/agents/agent_chooser.py

@@ -1,21 +1,22 @@
 from app.llm_model import get_gemini_2_5_flash_model
 from app.config import GEMINI_API_KEY
 from pydantic import BaseModel, Field
-from typing import Optional
+from typing import List, Optional
 import os
 from langchain_core.prompts import ChatPromptTemplate
 from langchain_litellm import ChatLiteLLM
 from app.log import get_logger
 from app.langfuse import get_langfuse_handler
 from app.util import get_session_id
-from sqlmodel import Session
+from sqlmodel import Session, select, col
 from app.agents.agent_memory import try_handle_profile, check_prompt_cache, save_memory
+from app.models import RagDocument
 
 
 class ChooserResult(BaseModel):
     action: str = Field(
         ...,
-        description="The action to take: 'car_hire_agent', 'weather_agent', 'generate_image_agent', 'marketing_agent', 'direct_answer', or 'unsupported'"
+        description="The action to take: 'car_hire_agent', 'weather_agent', 'generate_image_agent', 'marketing_agent', 'rag_ingest_agent', 'rag_query_agent', 'direct_answer', or 'unsupported'"
     )
     direct_response: Optional[str] = Field(
         None,
@@ -25,6 +26,18 @@ class ChooserResult(BaseModel):
 
 log = get_logger('agent-chooser')
 
+KNOWN_SOURCES_LIMIT = 20
+
+
+def get_known_rag_sources(session: Session, limit: int = KNOWN_SOURCES_LIMIT) -> List[str]:
+    """
+    Fetch a bounded list of previously ingested URLs so the router can recognize
+    questions that relate to content already added to the RAG knowledge base.
+    """
+    statement = select(RagDocument).order_by(col(RagDocument.fetched_at).desc()).limit(limit)
+    documents = session.exec(statement).all()
+    return [doc.url for doc in documents]
+
 
 def choose_agent(prompt: str, session: Optional[Session] = None, user_id: str = "default_user") -> ChooserResult:
     """
@@ -58,6 +71,12 @@ def choose_agent(prompt: str, session: Optional[Session] = None, user_id: str =
 
     structured_llm = llm.with_structured_output(ChooserResult)
 
+    known_sources = get_known_rag_sources(session) if session else []
+    known_sources_text = (
+        "\n".join(f"- {url}" for url in known_sources)
+        if known_sources else "(none yet)"
+    )
+
     # TODO use abstraction to build system prompt and agent discovery from app.agents.
     prompt_template = ChatPromptTemplate.from_messages([
         ("system", (
@@ -66,9 +85,13 @@ def choose_agent(prompt: str, session: Optional[Session] = None, user_id: str =
             "- If the user asks about the weather, temperature, rain, or weather forecast, set action to 'weather_agent'.\n"
             "- If the user asks to generate, draw, paint, create, or design an image, picture, photo, or graphic, set action to 'generate_image_agent'.\n"
             "- If the user asks to draft, generate, or write a marketing campaign, ad copy, marketing copy, or Facebook/Instagram/Google ad, set action to 'marketing_agent'.\n"
+            "- If the user asks to add, scrape, index, or learn a URL/website into the knowledge base (e.g. 'add https://... to rag knowledge base'), set action to 'rag_ingest_agent'.\n"
+            "- If the user asks a question that relates to one of the Known Knowledge Base Sources below (by domain, brand, or topic, even if not an exact match), or explicitly asks to look something up 'from the knowledge base' / 'from what you scraped', set action to 'rag_query_agent'.\n"
             "- If the user asks to remember a fact/preference about them, or asks what you know/remember about them (e.g. 'who am i', 'do you know my name'), set action to 'direct_answer' and write a helpful response in direct_response.\n"
             "- If the user asks a general knowledge question that you can answer directly (e.g. 'what is E=mc2'), set action to 'direct_answer' and write the answer in direct_response.\n"
-            "- If the user asks for anything else that requires a tool we do not have (e.g. booking a flight, ordering food, writing code), set action to 'unsupported' and set direct_response to exactly 'I can not do that at the moment'."
+            "- If the user asks for anything else that requires a tool we do not have (e.g. booking a flight, ordering food, writing code), set action to 'unsupported' and set direct_response to exactly 'I can not do that at the moment'.\n\n"
+            "Known Knowledge Base Sources (URLs already added via rag_ingest_agent):\n"
+            "{known_sources}"
         )),
         ("user", "{prompt}")
     ])
@@ -84,7 +107,7 @@ def choose_agent(prompt: str, session: Optional[Session] = None, user_id: str =
             }
         } if handler else {}
 
-        result = chain.invoke({"prompt": prompt}, config=config)
+        result = chain.invoke({"prompt": prompt, "known_sources": known_sources_text}, config=config)
 
         if result.action == "unsupported":
             result.direct_response = "I can not do that at the moment"