diff --git a/README.md b/README.md index ad5edce9..d50cdcac 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,214 @@ -# BYK-RAG (Retrieval-Augmented Generation Module) +# LLM Module -The **BYK-RAG Module** is part of the Burokratt ecosystem, designed to provide **retrieval-augmented generation (RAG)** capabilities for Estonian government digital services. It ensures reliable, multilingual, and compliant AI-powered responses by integrating with multiple LLM providers syncing with knowledge bases, and exposing flexible configuration and monitoring features for administrators. +The **LLM Module** is the LLM orchestration component of the +[Bürokratt](https://github.com/buerokratt) ecosystem, providing reliable, multilingual, and compliant +AI-powered responses for Estonian government digital services. It is a **multi-workflow orchestrator**: +a tool classifier inspects every user query and routes it to the most appropriate workflow — answering +from the knowledge base, calling backend services and APIs, handling conversation, or declining +gracefully when a request is out of scope. ---- +## Overview -## Features +Rather than treating every request as a single retrieval problem, the LLM Module classifies intent and +dispatches to one of several specialised workflows. Retrieval-Augmented Generation (RAG) is **one** of +these workflows — alongside Service, Context, API-Tool, and Out-of-Domain handling. All workflows run +over configurable, multi-provider LLMs, are protected by safety guardrails, and are fully traced for +cost and quality. -- **Configurable LLM Providers** - - Support for AWS Bedrock, Azure AI, Google Cloud, OpenAI, Anthropic, and self-hosted open-source LLMs. - - Admins can create "connections" and switch providers/models without downtime. - - Models searchable via dropdown with cache-enabled indicators. +### Key Features -- **Enhanced Security with RSA Encryption** - - LLM credentials encrypted with RSA-2048 asymmetric encryption before storage. - - GUI encrypts using public key; CronManager decrypts with private key. - - Additional security layer beyond HashiCorp Vault's encryption. +- **Multi-workflow orchestration** — a tool classifier routes each query to the Service, Context, + **RAG**, API-Tool, or Out-of-Domain (OOD) workflow using hybrid dense + sparse (BM25) search. +- **Configurable LLM providers** — Azure OpenAI, AWS Bedrock, Google Cloud, OpenAI, Anthropic, and + self-hosted models. Admins create "connections" and switch providers/models without downtime. +- **Grounded, cited answers** — RAG responses are restricted to Central Knowledge Base content, with + clear citations and an "I don't know" fallback when confidence is low. +- **Agentic API tool calling** — decomposes multi-intent queries and executes multi-endpoint API + workflows to fulfil actionable requests. +- **Secure credential management** — provider credentials stored in HashiCorp Vault with an additional + RSA-2048 encryption layer. +- **Safety guardrails** — NeMo Guardrails check input and output content with cost tracking. +- **Observability** — Langfuse for traces/cost analytics and Grafana/Loki for logs. -- **Knowledge Base Integration** - - Continuous sync with central knowledge base (CKB). - - Last sync timestamp displayed in UI. - - LLMs restricted to answering only from CKB content. - - "I don't know" payload returned when confidence is low. +## Architecture -- **Citations & Transparency** - - All responses are accompanied with **clear citations**. +The LLM Module sits behind the Ruuter API gateway and orchestrates retrieval, generation, and tool +calling over a set of supporting data stores (Qdrant, Redis, PostgreSQL/ClickHouse, MinIO) and +HashiCorp Vault for secrets. -- **Analytics & Monitoring** - - External **Langfuse dashboard** for API usage, inference trends, cost analysis, and performance logs. - - Agencies can configure cost alerts and view alerts via LLM Alerts UI. - - Logs integrated with **Grafana Loki**. +![LLM Module — System Context](./docs/images/LLM%20Module%20Context%20Diagram%20(Current).png) -### Storing Langfuse Secrets +For the full picture — container and component (C4) diagrams plus the request lifecycle — see +**[docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md)**. -1. **Generate API keys from Langfuse UI** (Settings → Project → API Keys) +## Tech Stack + +| Concern | Technology | +| --- | --- | +| Language / runtime | Python 3.12.10 | +| Package manager | [uv](https://docs.astral.sh/uv/) | +| API framework | FastAPI + Uvicorn (port `8100`) | +| LLM pipelines | DSPy | +| Safety | NeMo Guardrails | +| Vector database | Qdrant | +| Sessions & history | Redis | +| Analytics store | PostgreSQL + ClickHouse (Langfuse) | +| Object storage | MinIO (S3-compatible) | +| Secrets | HashiCorp Vault | +| Observability | Langfuse, Grafana, Loki | +| API gateway | Ruuter | + +## Quick Start + +### Prerequisites + +- Docker and Docker Compose +- [uv](https://docs.astral.sh/uv/) (for local development outside containers) + +### Run the stack -2. **Copy the script to vault container:** ```bash -docker cp store-langfuse-secrets.sh vault:/tmp/store-langfuse-secrets.sh +# Start the full stack (orchestration service + data stores + tooling) +docker compose up -d + +# Check the orchestration service health +curl http://localhost:8100/health ``` -3. **Execute the script with your API keys:** +### Environment configuration + +Configuration is supplied through environment files at the repository root: + +| File | Scope | +| --- | --- | +| `.env` | Shared infrastructure (storage, databases, Redis, Vault, feature flags) | +| `.env.llm_orchestration_service` | LLM Orchestration Service | +| `.env.gui` | Admin GUI | +| `.env.notification` | Notification server | + +Feature flags such as `TOOL_CLASSIFIER_ENABLED`, `SERVICE_WORKFLOW_ENABLED`, +`CONTEXT_WORKFLOW_ENABLED`, and `API_TOOL_CALLING_WORKFLOW_ENABLED` toggle individual workflows. + +## Components + +The orchestration service lives under `src/`. The main packages: + +| Component | Responsibility | Path | +| --- | --- | --- | +| Orchestration service & API | FastAPI app coordinating the pipeline | `src/llm_orchestration_service_api.py`, `src/llm_orchestration_service.py` | +| Tool classifier & workflows | Intent detection and per-workflow execution (Service / Context / RAG / API-Tool / OOD) | `src/tool_classifier/` | +| Contextual retrieval | Hybrid semantic + BM25 search with RRF fusion | `src/contextual_retrieval/` | +| Vector indexer | Document ingestion and embedding into Qdrant | `src/vector_indexer/` | +| API tool indexer | Indexing of API endpoints for tool calling | `src/api_tool_indexer/` | +| Intent data enrichment | LLM-generated enrichment of intent data | `src/intent_data_enrichment/` | +| Response generator | Grounded answer generation with citations | `src/response_generator/` | +| Prompt refiner | Query refinement for better retrieval | `src/prompt_refine_manager/` | +| Guardrails | NeMo Guardrails input/output safety | `src/guardrails/` | +| LLM configuration | Provider management, Vault credentials, feature flags | `src/llm_orchestrator_config/` | +| Optimization | DSPy-based tuning of guardrails, refiner, generator | `src/optimization/` | +| Utilities | Redis, sessions, rate limiting, streaming, cost, logging | `src/utils/` | + +## Core Workflows + +The tool classifier routes each query to one of: + +- **Service** — maps the query to a backend service/intent ([docs](./docs/TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md)) +- **Context** — greetings and conversation handling with Redis-backed history ([docs](./docs/CONTEXT_WORKFLOW_GREETING_DETECTION.md)) +- **RAG** — contextual retrieval then grounded generation ([docs](./docs/CONTEXTUAL_RETRIEVAL_FLOW.md)) +- **API-Tool** — agentic multi-endpoint API calling ([docs](./docs/API_TOOL_CALLING.md)) +- **OOD** — graceful fallback for out-of-domain queries + +Classification itself uses hybrid dense + sparse search — see +[docs/HYBRID_SEARCH_CLASSIFICATION.md](./docs/HYBRID_SEARCH_CLASSIFICATION.md). + +## Documentation + +The full documentation catalogue lives in **[docs/README.md](./docs/README.md)**, covering +architecture, retrieval, workflows, configuration/secrets, sessions, and the API reference. + +## Development + ```bash +# Install the pinned Python and dependencies +uv python install 3.12.10 +uv sync --frozen + +# Install pre-commit hooks +uv run pre-commit install + +# Run the test suite +uv run pytest tests/ -v +``` + +See **[CONTRIBUTING.md](./CONTRIBUTING.md)** for the full development workflow, tooling, and CI checks. + +## Deployment + +Several Docker Compose variants are provided for different environments: + +| File | Purpose | +| --- | --- | +| `docker-compose.yml` | Default full stack | +| `docker-compose-ec2.yml` | AWS EC2 deployment variant | +| `docker-compose-test.yml` | Integration testing | +| `docker-compose-eval.yml` | Evaluation / benchmarking | + +Kubernetes manifests and Helm charts are under [`kubernetes/`](./kubernetes), including +[`LANGFUSE_SETUP.md`](./kubernetes/LANGFUSE_SETUP.md) and +[`CONTAINER_REGISTRY_SETUP.md`](./kubernetes/CONTAINER_REGISTRY_SETUP.md). + +## API Reference + +HTTP endpoints for LLM connections, inference results, and chatbot inquiries are documented in +**[docs/API_REFERENCE.md](./docs/API_REFERENCE.md)**. + +## Configuration + +- **Environment files** — see [Environment configuration](#environment-configuration) above. +- **Feature flags** — `src/llm_orchestrator_config/feature_flags.py`. +- **Service configuration** — YAML configs under each module's `config/` directory (e.g. LLM + providers, contextual retrieval parameters, indexer settings, guardrails policies). +- **Secrets** — managed in HashiCorp Vault; see + [docs/LLM_CONFIG_VAULT_INTEGRATION.md](./docs/LLM_CONFIG_VAULT_INTEGRATION.md) and + [docs/VAULT_SETUP_AND_USAGE.md](./docs/VAULT_SETUP_AND_USAGE.md). + +### Storing Langfuse secrets + +Generate API keys in the Langfuse UI (**Settings → Project → API Keys**), then store them in Vault. + +For Docker Compose deployments, use the [`store-langfuse-secrets.sh`](./store-langfuse-secrets.sh) +script: + +```bash +# Copy the script into the vault container +docker cp store-langfuse-secrets.sh vault:/tmp/store-langfuse-secrets.sh + +# Run it with your Langfuse keys docker exec -e LANGFUSE_INIT_PROJECT_PUBLIC_KEY= \ -e LANGFUSE_INIT_PROJECT_SECRET_KEY= \ vault sh -c "chmod +x /tmp/store-langfuse-secrets.sh && /tmp/store-langfuse-secrets.sh" ``` + +For Kubernetes, see [kubernetes/LANGFUSE_SETUP.md](./kubernetes/LANGFUSE_SETUP.md). + +## Troubleshooting + +| Symptom | Where to look | +| --- | --- | +| Service unhealthy | `curl http://localhost:8100/health`; `docker compose ps`; `docker compose logs llm-orchestration-service` | +| Vault / credential errors | [docs/VAULT_SETUP_AND_USAGE.md](./docs/VAULT_SETUP_AND_USAGE.md) and [docs/VAULT_SECURITY_ARCHITECTURE.md](./docs/VAULT_SECURITY_ARCHITECTURE.md) | +| Missing conversation history | Redis connectivity (`REDIS_HOST`/`REDIS_AUTH`); [docs/REDIS_SESSION_STORE.md](./docs/REDIS_SESSION_STORE.md) | +| Retrieval returns nothing | Qdrant availability (port `6333`) and that indexing has run | +| Logs & traces | Grafana/Loki for logs; Langfuse for request traces and cost | + +## License + +This project is licensed under the terms in the [LICENSE](./LICENSE) file. + +## Links + +- [Bürokratt project](https://github.com/buerokratt) +- [Architecture](./docs/ARCHITECTURE.md) +- [Documentation index](./docs/README.md) +- [API reference](./docs/API_REFERENCE.md) +- [Contributing](./CONTRIBUTING.md) diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md new file mode 100644 index 00000000..9aed52cc --- /dev/null +++ b/docs/API_REFERENCE.md @@ -0,0 +1,1760 @@ +# API Reference + +This document is the consolidated HTTP API reference for the LLM Module. It covers the **LLM +Connections** management endpoints, the **Inference Results** storage/retrieval endpoints, and the +chatbot **Inquiry** endpoint exposed to the LLM Orchestration Service. + +> Routing note: the public-facing paths below are served through the Ruuter API gateway +> (`ruuter-private` / `ruuter-public`), which proxies to the LLM Orchestration Service. See +> [ARCHITECTURE.md](./ARCHITECTURE.md) for how requests flow through the system. + +## Contents + +- [LLM Connections API](#llm-connections-api-endpoints) +- [Inference Results API](#inference-results-api-endpoints) + +--- + +## LLM Connections API Endpoints + +### Base URL +``` +/ruuter-private/llm/connections +``` + +--- + +## 1. Create LLM Connection + +### Endpoint +```http +POST /ruuter-private/llm/connections/create +``` + +### Request Body +```json +{ + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "deploymentEnvironment": "Testing", + // Azure credentials (optional) + "deploymentName": "my-deployment", + "targetUri": "https://my-endpoint.azure.com", + "apiKey": "azure-api-key", + // AWS Bedrock credentials (optional) + "secretKey": "aws-secret-key", + "accessKey": "aws-access-key", + // Embedding model credentials (optional) + "embeddingModelApiKey": "embedding-api-key" +} +``` + +### Response (201 Created) +```json +{ + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "usedBudget": 0.00, + "deploymentEnvironment": "Testing", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + // Azure credentials (if provided) + "deploymentName": "my-deployment", + "targetUri": "https://my-endpoint.azure.com", + "apiKey": "azure-api-key", + // AWS Bedrock credentials (if provided) + "secretKey": "aws-secret-key", + "accessKey": "aws-access-key", + // Embedding model credentials (if provided) + "embeddingModelApiKey": "embedding-api-key" +} +``` + +--- + +## 2. Update LLM Connection + +### Endpoint +```http +POST /ruuter-private/llm/connections/update +``` + +### Request Body +```json +{ + "connectionId": 1, + "llmPlatform": "Azure AI", + "llmModel": "GPT-4o-mini", + "embeddingPlatform": "Azure AI", + "embeddingModel": "text-embedding-ada-002", + "monthlyBudget": 2000.00, + "deploymentEnvironment": "Production", + // Azure credentials (optional) + "deploymentName": "updated-deployment", + "targetUri": "https://updated-endpoint.azure.com", + "apiKey": "updated-azure-api-key", + // AWS Bedrock credentials (optional) + "secretKey": "updated-aws-secret-key", + "accessKey": "updated-aws-access-key", + // Embedding model credentials (optional) + "embeddingModelApiKey": "updated-embedding-api-key" +} +``` + +### Response (200 OK) +```json +{ + "id": 1, + "llmPlatform": "Azure AI", + "llmModel": "GPT-4o-mini", + "embeddingPlatform": "Azure AI", + "embeddingModel": "text-embedding-ada-002", + "monthlyBudget": 2000.00, + "usedBudget": 150.75, + "deploymentEnvironment": "Production", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + // Azure credentials (if provided) + "deploymentName": "updated-deployment", + "targetUri": "https://updated-endpoint.azure.com", + "apiKey": "updated-azure-api-key", + // AWS Bedrock credentials (if provided) + "secretKey": "updated-aws-secret-key", + "accessKey": "updated-aws-access-key", + // Embedding model credentials (if provided) + "embeddingModelApiKey": "updated-embedding-api-key" +} +``` + +--- + +## 3. Get LLM Connections (Paginated List) + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/list +``` + +### Request Body +```json +{ + "page": 1, + "page_size": 10, + "sorting": "created_at desc" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | Default | +|-----------|------|----------|-------------|---------| +| `page` | number | No | Page number (1-based) | 1 | +| `page_size` | number | No | Number of items per page | 10 | +| `sorting` | string | No | Sorting criteria | "created_at desc" | + +### Sorting Options +- `llm_platform asc/desc` +- `llm_model asc/desc` +- `embedding_platform asc/desc` +- `embedding_model asc/desc` +- `monthly_budget asc/desc` +- `environment asc/desc` +- `status asc/desc` +- `created_at asc/desc` +- `updated_at asc/desc` + +### Response (200 OK) +```json +[ + { + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "environment": "Testing", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + "updatedAt": "2025-09-02T10:15:30.000Z", + "totalPages": 3 + }, + { + "id": 2, + "llmPlatform": "Azure AI", + "llmModel": "GPT-4o-mini", + "embeddingPlatform": "Azure AI", + "embeddingModel": "Ada-200-1", + "monthlyBudget": 2000.00, + "environment": "Production", + "status": "active", + "createdAt": "2025-09-02T09:30:15.000Z", + "updatedAt": "2025-09-02T11:00:00.000Z", + "totalPages": 3 + } +] +``` + +--- + +## 4. Get Single LLM Connection + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/get +``` + +### Request Body +```json +{ + "connection_id": 1 +} +``` + +### Response (200 OK) +```json +{ + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "environment": "Testing", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + "updatedAt": "2025-09-02T10:15:30.000Z" +} +``` + +### Response (404 Not Found) +```json +"error: connection not found" +``` + +--- + +## 5. Add New LLM Connection + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/add +``` + +### Request Body +```json +{ + "llm_platform": "OpenAI", + "llm_model": "GPT-4o", + "embedding_platform": "OpenAI", + "embedding_model": "text-embedding-3-small", + "monthly_budget": 1000.00, + "environment": "Testing" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `llm_platform` | string | Yes | LLM platform (e.g., "Azure AI", "OpenAI") | +| `llm_model` | string | Yes | LLM model (e.g., "GPT-4o") | +| `embedding_platform` | string | Yes | Embedding platform | +| `embedding_model` | string | Yes | Embedding model | +| `monthly_budget` | number | Yes | Monthly budget amount | +| `environment` | string | Yes | "Testing" or "Production" | + +### Response (200 OK) +```json +{ + "id": 3, + "llm_platform": "OpenAI", + "llm_model": "GPT-4o", + "embedding_platform": "OpenAI", + "embedding_model": "text-embedding-3-small", + "monthly_budget": 1000.00, + "environment": "Testing", + "status": "active", + "created_at": "2025-09-02T12:00:00.000Z", + "updated_at": "2025-09-02T12:00:00.000Z" +} +``` + +### Response (400 Bad Request) +```json +"error: environment must be 'Testing' or 'Production'" +``` + +--- + +## 6. Update LLM Connection + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/edit +``` + +### Request Body +```json +{ + "connection_id": 1, + "llm_platform": "Azure AI", + "llm_model": "GPT-4o-mini", + "embedding_platform": "Azure AI", + "embedding_model": "Ada-200-1", + "monthly_budget": 2000.00, + "environment": "Production" +} +``` + +### Response (200 OK) +```json +{ + "id": 1, + "llm_platform": "Azure AI", + "llm_model": "GPT-4o-mini", + "embedding_platform": "Azure AI", + "embedding_model": "Ada-200-1", + "monthly_budget": 2000.00, + "environment": "Production", + "status": "active", + "created_at": "2025-09-02T10:15:30.000Z", + "updated_at": "2025-09-02T12:30:00.000Z" +} +``` + +### Response (404 Not Found) +```json +"error: connection not found" +``` + +--- + +## 7. Delete LLM Connection + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/delete +``` + +### Request Body +```json +{ + "connection_id": 1 +} +``` + +### Response (200 OK) +```json +"LLM connection deleted successfully" +``` + +### Response (404 Not Found) +```json +"error: connection not found" +``` + +--- + +## 4. List All LLM Connections + +### Endpoint +```http +GET /ruuter-private/llm/connections/list +``` + +### Query Parameters (Optional for filtering) +| Parameter | Type | Description | +|-----------|------|-------------| +| `llmPlatform` | `string` | Filter by LLM platform | +| `llmModel` | `string` | Filter by LLM model | +| `deploymentEnvironment` | `string` | Filter by environment (Testing / Production) | +| `pageNumber` | `number` | Page number (1-based) | +| `pageSize` | `number` | Number of items per page | +| `sortBy` | `string` | Field to sort by | +| `sortOrder` | `string` | Sort order: 'asc' or 'desc' | + +### Example Request +```http +GET /ruuter-private/llm/connections/list?llmPlatform=OpenAI&deploymentEnvironment=Testing&model=GPT4 +``` + +--- + +## 5. Get Production LLM Connection (with filters) + +### Endpoint +```http +GET /ruuter-private/llm/connections/production +``` + +### Query Parameters (Optional for filtering) +| Parameter | Type | Description | +|-----------|------|-------------| +| `llmPlatform` | `string` | Filter by LLM platform | +| `llmModel` | `string` | Filter by LLM model | +| `embeddingPlatform` | `string` | Filter by embedding platform | +| `embeddingModel` | `string` | Filter by embedding model | +| `connectionStatus` | `string` | Filter by connection status | +| `sortBy` | `string` | Field to sort by | +| `sortOrder` | `string` | Sort order: 'asc' or 'desc' | + +### Example Request +```http +GET /ruuter-private/llm/connections/production?llmPlatform=OpenAI&connectionStatus=active +``` + +### Response (200 OK) +```json +[ + { + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "deploymentEnvironment": "Testing", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + "updatedAt": "2025-09-02T10:15:30.000Z" + } +] +``` + +--- + +## 5. Get Single LLM Connection + +### Endpoint +```http +GET /ruuter-private/llm/connections/overview +``` + +### Response (200 OK) +```json +{ + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "deploymentEnvironment": "Testing", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + "updatedAt": "2025-09-02T10:15:30.000Z" +} +``` + +--- + +## 6. Check if LLM Connection Exists + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/exists +``` + +### Request Body +```json +{ "connection_id": 1 } +``` + +### Response (200 OK) +```json +"true" +``` +or +```json +"false" +``` + +--- + +## 7. Update LLM Connection Status + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/update-status +``` + +### Request Body +```json +{ + "connection_id": 1, + "connection_status": "inactive" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `connection_id` | number | Yes | LLM connection ID | +| `connection_status` | string | Yes | `"active"` or `"inactive"` | + +### Response (200 OK) +Returns the updated connection object. + +### Response (400 Bad Request) +```json +"error: connection_status must be 'active' or 'inactive'" +``` + +### Response (404 Not Found) +```json +"error: connection not found" +``` + +--- + +## 8. List LLM Connections — GET (Paginated, with Filters) + +### Endpoint +```http +GET /ruuter-private/rag-search/llm-connections/list +``` + +### Query Parameters +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageNumber` | number | No | `1` | Page number (1-based) | +| `pageSize` | number | No | `10` | Items per page (1–100) | +| `sortBy` | string | No | `"created_at"` | Field to sort by | +| `sortOrder` | string | No | `"desc"` | `"asc"` or `"desc"` | +| `llmPlatform` | string | No | `""` | Filter by LLM platform | +| `llmModel` | string | No | `""` | Filter by LLM model | +| `environment` | string | No | `""` | Filter by environment | + +### Example Request +```http +GET /ruuter-private/rag-search/llm-connections/list?pageNumber=1&pageSize=10&llmPlatform=OpenAI +``` + +### Response (200 OK) +```json +[ + { + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "environment": "Testing", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + "updatedAt": "2025-09-02T10:15:30.000Z", + "totalPages": 3 + } +] +``` + +### Response (400 Bad Request) +```json +"Page number must be greater than 0" +``` + +--- + +## 9. List All LLM Connections — GET (Paginated, with Filters) + +### Endpoint +```http +GET /ruuter-private/rag-search/llm-connections/all +``` + +Same as endpoint 8 above but queries all connections regardless of status. Accepts the same query parameters. + +--- + +## 10. Get Production LLM Connection — GET (with Filters) + +### Endpoint +```http +GET /ruuter-private/rag-search/llm-connections/production +``` + +### Query Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `llmPlatform` | string | No | Filter by LLM platform | +| `llmModel` | string | No | Filter by LLM model | +| `embeddingPlatform` | string | No | Filter by embedding platform | +| `embeddingModel` | string | No | Filter by embedding model | +| `connectionStatus` | string | No | Filter by connection status | +| `sortBy` | string | No | Field to sort by (default: `"created_at"`) | +| `sortOrder` | string | No | `"asc"` or `"desc"` (default: `"desc"`) | + +### Example Request +```http +GET /ruuter-private/rag-search/llm-connections/production?connectionStatus=active +``` + +### Response (200 OK) +```json +[ + { + "id": 1, + "llmPlatform": "OpenAI", + "llmModel": "GPT-4o", + "embeddingPlatform": "OpenAI", + "embeddingModel": "text-embedding-3-small", + "monthlyBudget": 1000.00, + "environment": "Production", + "status": "active", + "createdAt": "2025-09-02T10:15:30.000Z", + "updatedAt": "2025-09-02T10:15:30.000Z" + } +] +``` + +--- + +## 11. Update Used Budget for a Connection + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/cost/update +``` + +Adds `usage` to the connection's current `used_budget`. If `disconnectOnBudgetExceed` is set and the stop threshold is reached, the connection is automatically deactivated. + +### Request Body +```json +{ + "connection_id": 1, + "usage": 12.50 +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `connection_id` | number | Yes | LLM connection ID | +| `usage` | number | Yes | Amount to add to `used_budget` (≥ 0) | + +### Response (200 OK) — within budget +```json +{ + "data": { "id": 1, "usedBudget": 162.50, "monthlyBudget": 1000.00 }, + "budgetExceeded": false, + "message": "Used budget updated successfully", + "operationSuccess": true, + "statusCode": 200 +} +``` + +### Response (200 OK) — budget exceeded, connection deactivated +```json +{ + "data": { "id": 1, "usedBudget": 1005.00, "status": "inactive" }, + "budgetExceeded": true, + "message": "Used budget updated successfully. Connection deactivated due to budget threshold exceeded.", + "operationSuccess": true, + "statusCode": 200 +} +``` + +### Response (400 Bad Request) +```json +"error: connection_id and usage (>= 0) are required" +``` + +### Response (404 Not Found) +```json +"error: connection not found" +``` + +--- + +## 12. Check Budget Usage + +### Endpoint +```http +POST /ruuter-private/rag-search/llm-connections/usage/check +``` + +Returns whether the connection's budget is within the stop threshold, exceeded (not disconnected), or exceeded with disconnection. + +### Request Body +```json +{ "connection_id": 1 } +``` + +### Response (200 OK) — within budget +```json +{ + "isBudgetExceed": false, + "isLLMConnectionDisconnected": false +} +``` + +### Response (200 OK) — exceeded, not disconnected +```json +{ + "isBudgetExceed": true, + "isLLMConnectionDisconnected": false +} +``` + +### Response (200 OK) — exceeded and disconnected +```json +{ + "isBudgetExceed": true, + "isLLMConnectionDisconnected": true +} +``` + +### Response (404 Not Found) +```json +"Connection not found" +``` + +--- + +## 13. Check Budget Thresholds for Production Connection + +### Endpoint +```http +GET /ruuter-private/rag-search/llm-connections/cost/check +``` + +Returns warn/stop threshold status for the active production connection. + +### Response (200 OK) +```json +{ + "data": { + "id": 1, + "monthlyBudget": 1000.00, + "usedBudget": 620.00, + "warnBudgetThreshold": 70, + "stopBudgetThreshold": 90 + }, + "used_budget_percentage": 62.0, + "exceeded_stop_budget": false, + "exceeded_warn_budget": false +} +``` + +### Response (404 Not Found) +```json +"No production LLM connection found" +``` + +--- + +## 14. Reset Used Budget for All Connections + +### Endpoint +```http +POST /ruuter-public/rag-search/llm-connections/cost/reset +``` + +Resets `used_budget` to `0` for all LLM connections. Typically called by a scheduled job at the start of each billing period. + +### Request Body +None required. + +### Response (200 OK) +```json +{ + "message": "Used budget reset to 0 successfully for all connections", + "totalConnections": "5", + "operationSuccess": true, + "statusCode": 200 +} +``` + +### Response (500 Internal Server Error) +```json +"error: failed to reset used budget" +``` + +--- +# Inference Results API Endpoints + +## Base URL +``` +/ruuter-private/inference/results +``` + +--- + +## 1. Store Test Inference Result + +### Endpoint +```http +POST /ruuter-private/inference/results/test/store +``` + +### Request Body +```json +{ + "llm_connection_id": 1, + "user_question": "What are the benefits of using LLMs?", + "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `llm_connection_id` | number | Yes | ID of the LLM connection | +| `user_question` | string | Yes | User's raw question/input | +| `final_answer` | string | Yes | LLM's final generated answer | + +### Response (200 OK) +```json +{ + "data": { + "id": 10, + "llm_connection_id": 1, + "chat_id": null, + "user_question": "What are the benefits of using LLMs?", + "refined_questions": null, + "conversation_history": null, + "ranked_chunks": null, + "embedding_scores": null, + "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", + "environment": "testing", + "created_at": "2025-09-25T12:15:00.000Z" + }, + "operationSuccess": true, + "statusCode": 200 +} +``` + +### Response (400 Bad Request) +```json +{ + "data": "[]", + "operationSuccess": false, + "statusCode": 400 +} +``` + +### Response (404 Not Found) +```json +"error: LLM connection not found" +``` + +--- + +## 2. Store Production Inference Result + +### Endpoint +```http +POST /ruuter-private/inference/results/production/store +``` + +### Request Body +```json +{ + "chat_id": "chat-12345", + "user_question": "What are the benefits of using LLMs?", + "refined_questions": [ + "How do LLMs improve productivity?", + "What are practical use cases of LLMs?" + ], + "conversation_history": [ + { "role": "user", "content": "Hello" }, + { "role": "assistant", "content": "Hi! How can I help you?" } + ], + "ranked_chunks": [ + { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, + { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } + ], + "embedding_scores": [0.92, 0.85, 0.78], + "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `chat_id` | string | No | Optional chat session ID | +| `user_question` | string | Yes | User's raw question/input | +| `refined_questions` | object | No | List of refined questions (LLM-generated) | +| `conversation_history` | object | No | Prior messages array of {role, content} | +| `ranked_chunks` | object | No | Retrieved chunks ranked with metadata | +| `embedding_scores` | object | No | Distance scores for each chunk | +| `final_answer` | string | Yes | LLM's final generated answer | + +### Response (200 OK) +```json +{ + "data": { + "id": 15, + "llm_connection_id": null, + "chat_id": "chat-12345", + "user_question": "What are the benefits of using LLMs?", + "refined_questions": [ + "How do LLMs improve productivity?", + "What are practical use cases of LLMs?" + ], + "conversation_history": [ + { "role": "user", "content": "Hello" }, + { "role": "assistant", "content": "Hi! How can I help you?" } + ], + "ranked_chunks": [ + { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, + { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } + ], + "embedding_scores": [0.92, 0.85, 0.78], + "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", + "environment": "production", + "created_at": "2025-09-25T12:15:00.000Z" + }, + "operationSuccess": true, + "statusCode": 200 +} +``` + +### Response (400 Bad Request) +```json +{ + "data": "[]", + "operationSuccess": false, + "statusCode": 400 +} +``` + +--- + +## 3. View/get Inference Result + +### Endpoint +```http +POST /ruuter-private/inference/results/test/store +``` + +### Request Body +```json +{ + "llmConnectionId": 1, + "userQuestion": "What are the benefits of using LLMs?", + "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." +} +``` + +### Response (201 Created) +```json +{ + "data": { + "id": 15, + "llmConnectionId": 1, + "userQuestion": "What are the benefits of using LLMs?", + "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", + "environment": "testing", + "createdAt": "2025-09-25T10:15:30.000Z" + }, + "operationSuccess": true, + "statusCode": 200 +} +``` + +## 4. Inquiry from chatbot to llm orchestration service + +### Endpoint +```http +POST /ruuter-private/inference/results/production/store +``` + +### Request Body +```json +{ + "llmConnectionId": 1, + "chatId": "chat-session-12345", + "userQuestion": "What are the benefits of using LLMs?", + "refinedQuestions": [ + "How do LLMs improve productivity?", + "What are practical use cases of LLMs?" + ], + "conversationHistory": [ + { "role": "user", "content": "Hello" }, + { "role": "assistant", "content": "Hi! How can I help you?" } + ], + "rankedChunks": [ + { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, + { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } + ], + "embeddingScores": { + "chunk_1": 0.92, + "chunk_2": 0.85 + }, + "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." +} +``` + +### Response (201 Created) +```json +{ + "id": 20, + "llmConnectionId": 1, + "chatId": "chat-session-12345", + "userQuestion": "What are the benefits of using LLMs?", + "refinedQuestions": [ + "How do LLMs improve productivity?", + "What are practical use cases of LLMs?" + ], + "conversationHistory": [ + { "role": "user", "content": "Hello" }, + { "role": "assistant", "content": "Hi! How can I help you?" } + ], + "rankedChunks": [ + { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, + { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } + ], + "embeddingScores": { + "chunk_1": 0.92, + "chunk_2": 0.85 + }, + "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", + "environment": "production", + "createdAt": "2025-09-25T10:15:30.000Z" +} +``` + +--- + +## 5. Production Inference + +### Endpoint +```http +POST /ruuter-private/rag-search/inference/production +``` + +Validates the production connection's budget then proxies the request to the LLM Orchestration Service. + +### Request Body +```json +{ + "chatId": "chat-session-123", + "message": "What are the benefits of using LLMs?", + "authorId": "user-456", + "conversationHistory": [ + { "role": "user", "content": "Hello" }, + { "role": "assistant", "content": "Hi! How can I help you?" } + ], + "url": "https://example.com/context" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `chatId` | string | Yes | Chat session ID | +| `message` | string | Yes | User message | +| `authorId` | string | Yes | Author ID | +| `conversationHistory` | array | No | Prior `{role, content}` messages | +| `url` | string | No | URL reference | + +### Response (200 OK) +Proxied response from the LLM Orchestration Service. + +### Response (400 Bad Request) — connection disconnected due to budget +```json +{ + "chatId": "chat-session-123", + "content": "The LLM connection is currently unavailable. Your request couldn't be processed. Please retry shortly.", + "status": 400 +} +``` + +### Response (404 Not Found) +```json +"No production connection found" +``` + +--- + +## 6. Test Inference + +### Endpoint +```http +POST /ruuter-private/rag-search/inference/test +``` + +Validates a specific connection's budget then calls the LLM Orchestration Service `/test` endpoint. + +### Request Body +```json +{ + "connectionId": "1", + "message": "What are the benefits of using LLMs?" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `connectionId` | string | Yes | Connection ID to test against | +| `message` | string | Yes | User message | + +### Response (200 OK) +Proxied response from the LLM Orchestration Service `/test` endpoint. + +### Response (400 Bad Request) — connection disconnected due to budget +```json +{ + "connectionId": "1", + "content": "The LLM connection is currently unavailable. Your request couldn't be processed. Please retry shortly.", + "status": 400 +} +``` + +### Response (404 Not Found) +```json +"No test connection found" +``` + +--- + +## 7. View Inference Result (Mock) + +### Endpoint +```http +POST /ruuter-private/rag-search/inference/results/view +``` + +Returns a mock inference response for testing purposes. + +### Request Body +```json +{ + "llmConnectionId": 1, + "message": "What services are available?" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `llmConnectionId` | number | Yes | LLM connection ID | +| `message` | string | Yes | User message/question | + +### Response (200 OK) +```json +{ + "chatId": 10, + "llmServiceActive": true, + "questionOutOfLlmScope": true, + "content": "Random answer with citations\n - https://gov.ee/sample1,\n - https://gov.ee/sample1" +} +``` + +### Response (400 Bad Request) +```json +"llmConnectionId and message are required" +``` + +--- + +## 8. Store Inference Result (Public) + +### Endpoint +```http +POST /ruuter-public/rag-search/inference/results/store +``` + +Public variant of the inference result store. Accepts the same fields as the private store endpoints, plus `environment` and `vault_uuid`. + +### Request Body +```json +{ + "user_question": "What are the benefits of using LLMs?", + "final_answer": "LLMs can improve productivity...", + "chat_id": "chat-12345", + "environment": "production", + "vault_uuid": "550e8400-e29b-41d4-a716-446655440000", + "refined_questions": ["How do LLMs improve productivity?"], + "conversation_history": [{ "role": "user", "content": "Hello" }], + "ranked_chunks": [{ "id": "chunk_1", "content": "...", "rank": 1 }], + "embedding_scores": [0.92, 0.85] +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `user_question` | string | Yes | User's raw question/input | +| `final_answer` | string | Yes | LLM's final generated answer | +| `chat_id` | string | No | Chat session ID | +| `environment` | string | No | Environment identifier | +| `vault_uuid` | string | No | Vault UUID for the LLM connection | +| `refined_questions` | object | No | List of refined questions | +| `conversation_history` | object | No | Prior `{role, content}` messages | +| `ranked_chunks` | object | No | Retrieved chunks ranked with metadata | +| `embedding_scores` | object | No | Distance scores for each chunk | + +### Response (200 OK) +```json +{ + "data": { "id": 20, "user_question": "...", "final_answer": "...", "environment": "production" }, + "operationSuccess": true, + "statusCode": 200 +} +``` + +### Response (400 Bad Request) +```json +{ + "data": "[]", + "operationSuccess": false, + "statusCode": 400 +} +``` + +--- + +# LLM Platforms & Models API Endpoints + +## Base URL +``` +/ruuter-private/rag-search +``` + +--- + +## 1. Get LLM Platforms + +### Endpoint +```http +GET /ruuter-private/rag-search/llm/platforms +``` + +Returns all active LLM platforms. + +### Response (200 OK) +```json +[ + { "id": 1, "value": "openai", "label": "OpenAI" }, + { "id": 2, "value": "azure", "label": "Azure AI" }, + { "id": 3, "value": "aws", "label": "AWS Bedrock" } +] +``` + +--- + +## 2. Get LLM Models by Platform + +### Endpoint +```http +GET /ruuter-private/rag-search/llm/models +``` + +### Query Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `platform_key` | string | Yes | Platform key to filter models (e.g. `"openai"`) | + +### Example Request +```http +GET /ruuter-private/rag-search/llm/models?platform_key=openai +``` + +### Response (200 OK) +```json +[ + { "id": 1, "value": "gpt-4o", "label": "GPT-4o", "platform_id": 1, "platform_key": "openai", "platform_name": "OpenAI" }, + { "id": 2, "value": "gpt-4o-mini", "label": "GPT-4o-mini", "platform_id": 1, "platform_key": "openai", "platform_name": "OpenAI" } +] +``` + +--- + +## 3. Get All LLM Models + +### Endpoint +```http +GET /ruuter-private/rag-search/llm/models-list +``` + +Returns all LLM models with no platform filter. + +### Response (200 OK) +```json +[ + { "id": 1, "platform_id": 1, "value": "gpt-4o", "label": "GPT-4o" }, + { "id": 2, "platform_id": 1, "value": "gpt-4o-mini", "label": "GPT-4o-mini" } +] +``` + +--- + +## 4. Get Embedding Platforms + +### Endpoint +```http +GET /ruuter-private/rag-search/embedding/platforms +``` + +Returns all active embedding platforms. + +### Response (200 OK) +```json +[ + { "id": 1, "value": "openai", "label": "OpenAI" }, + { "id": 2, "value": "azure", "label": "Azure AI" } +] +``` + +--- + +## 5. Get Embedding Models by Platform + +### Endpoint +```http +GET /ruuter-private/rag-search/embedding/models +``` + +### Query Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `embedding_platform_key` | string | Yes | Platform key to filter models | + +### Example Request +```http +GET /ruuter-private/rag-search/embedding/models?embedding_platform_key=openai +``` + +### Response (200 OK) +```json +[ + { "id": 1, "value": "text-embedding-3-small", "label": "text-embedding-3-small", "platform_id": 1, "platform_key": "openai", "platform_name": "OpenAI" }, + { "id": 2, "value": "text-embedding-ada-002", "label": "text-embedding-ada-002", "platform_id": 1, "platform_key": "openai", "platform_name": "OpenAI" } +] +``` + +--- + +# Prompt Configuration API Endpoints + +## Base URL +``` +/ruuter-private/rag-search/prompt-configuration +``` + +--- + +## 1. Get Prompt Configuration + +### Endpoint +```http +GET /ruuter-private/rag-search/prompt-configuration/get +``` + +Returns the active custom prompt configuration. Returns an empty array if none is configured. + +### Response (200 OK) +```json +[ + { + "id": 1, + "prompt": "You are a helpful assistant for government services...", + "created_at": "2025-09-02T10:15:30.000Z", + "updated_at": "2025-09-02T12:30:00.000Z" + } +] +``` + +--- + +## 2. Save Prompt Configuration + +### Endpoint +```http +POST /ruuter-private/rag-search/prompt-configuration/save +``` + +Upserts the prompt configuration (inserts if none exists, updates otherwise). Also triggers an LLM cache refresh. + +### Request Body +```json +{ + "prompt": "You are a helpful assistant for government services. Answer questions accurately and concisely." +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `prompt` | string | Yes | Prompt text to save | + +### Response (200 OK) +Returns the saved prompt configuration object. + +```json +{ + "id": 1, + "prompt": "You are a helpful assistant for government services. Answer questions accurately and concisely.", + "updated_at": "2025-09-25T12:00:00.000Z" +} +``` + +--- + +# Vault Secrets API Endpoints + +## Base URL +``` +/ruuter-private/rag-search/vault/secret +``` + +--- + +## 1. Create Vault Secret + +### Endpoint +```http +POST /ruuter-private/rag-search/vault/secret/create +``` + +Stores LLM connection credentials in Vault via CronManager. Supported platforms: `"aws"`, `"azure"`. + +### Request Body (AWS) +```json +{ + "vaultUuid": "550e8400-e29b-41d4-a716-446655440000", + "llmPlatform": "aws", + "llmModel": ["claude-3-sonnet"], + "secretKey": "aws-secret-key", + "accessKey": "aws-access-key", + "embeddingModel": "amazon.titan-embed-text-v1", + "embeddingPlatform": "aws", + "embeddingAccessKey": "embed-access-key", + "embeddingSecretKey": "embed-secret-key", + "deploymentEnvironment": "Production" +} +``` + +### Request Body (Azure) +```json +{ + "vaultUuid": "550e8400-e29b-41d4-a716-446655440000", + "llmPlatform": "azure", + "llmModel": ["gpt-4o"], + "deploymentName": "my-deployment", + "targetUrl": "https://my-endpoint.azure.com", + "apiKey": "azure-api-key", + "embeddingModel": "text-embedding-ada-002", + "embeddingPlatform": "azure", + "embeddingDeploymentName": "embed-deployment", + "embeddingTargetUri": "https://embed-endpoint.azure.com", + "embeddingAzureApiKey": "embed-azure-api-key", + "deploymentEnvironment": "Production" +} +``` + +### Request Parameters +| Parameter | Type | Platform | Description | +|-----------|------|----------|-------------| +| `vaultUuid` | string | Both | Stable UUID for the vault path | +| `llmPlatform` | string | Both | `"aws"` or `"azure"` | +| `llmModel` | array | Both | LLM model identifier(s) | +| `deploymentEnvironment` | string | Both | Deployment environment | +| `embeddingModel` | string | Both | Embedding model identifier | +| `embeddingPlatform` | string | Both | Embedding platform | +| `secretKey` | string | AWS | AWS secret key | +| `accessKey` | string | AWS | AWS access key | +| `embeddingAccessKey` | string | AWS | Embedding AWS access key | +| `embeddingSecretKey` | string | AWS | Embedding AWS secret key | +| `deploymentName` | string | Azure | Azure deployment name | +| `targetUrl` | string | Azure | Azure endpoint URL | +| `apiKey` | string | Azure | Azure API key | +| `embeddingDeploymentName` | string | Azure | Embedding Azure deployment name | +| `embeddingTargetUri` | string | Azure | Embedding Azure endpoint URI | +| `embeddingAzureApiKey` | string | Azure | Embedding Azure API key | + +### Response (200 OK) — AWS +```json +"Executed cron manager successfully to store aws secrets" +``` + +### Response (200 OK) — Azure +```json +"Executed cron manager successfully to store azure secrets" +``` + +### Response (400 Bad Request) +```json +{ + "message": "Platform not supported", + "operationSuccessful": false, + "statusCode": 400 +} +``` + +--- + +## 2. Delete Vault Secret + +### Endpoint +```http +POST /ruuter-private/rag-search/vault/secret/delete +``` + +Removes LLM connection credentials from Vault via CronManager. + +### Request Body +```json +{ + "vaultUuid": "550e8400-e29b-41d4-a716-446655440000", + "llmPlatform": "azure", + "llmModel": "gpt-4o", + "embeddingModel": "text-embedding-ada-002", + "embeddingPlatform": "azure", + "deploymentEnvironment": "Production" +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `vaultUuid` | string | Yes | Vault UUID of the connection | +| `llmPlatform` | string | Yes | LLM platform | +| `llmModel` | string | Yes | LLM model identifier | +| `embeddingModel` | string | Yes | Embedding model identifier | +| `embeddingPlatform` | string | Yes | Embedding platform | +| `deploymentEnvironment` | string | Yes | Deployment environment | + +### Response (200 OK) +```json +"Executed cron manager successfully to delete secrets from vault" +``` + +### Response (404 Not Found) +```json +{ + "message": "Connection not found with the provided vaultUuid", + "operationSuccessful": false, + "statusCode": 404 +} +``` + +--- + +# Data Sync & Services API Endpoints + +## Base URL +``` +/ruuter-public/rag-search +``` + +--- + +## 1. Get Services (for Intent Detection) + +### Endpoint +```http +GET /ruuter-public/rag-search/services/get-services +``` + +Returns all active services if the count is ≤ 10. If count > 10, signals the caller to use semantic search instead. + +### Response (200 OK) — ≤ 10 services +```json +{ + "use_semantic_search": false, + "service_count": 5, + "services": [ + { "id": "svc-1", "name": "Pension Application", "description": "..." } + ] +} +``` + +### Response (200 OK) — > 10 services +```json +{ + "use_semantic_search": true, + "service_count": 23, + "message": "Service count exceeds threshold - use semantic search" +} +``` + +--- + +## 2. Resync Data from KB + +### Endpoint +```http +POST /ruuter-public/rag-search/data/update +``` + +Fetches the latest agency data from CKB, compares the data hash, and if changed triggers vector re-indexing via CronManager. + +### Request Body +None required. + +### Response (200 OK) — sync initiated +```json +{ + "message": "Data synchronization initiated successfully", + "operationSuccessful": true +} +``` + +### Response (200 OK) — already up to date +```json +{ + "success": true, + "message": "No sync required - data is up to date" +} +``` + +### Response (400 Bad Request) +```json +{ + "message": "CKB service returned an error - data synchronization aborted", + "operationSuccessful": false, + "error": "CKB_ERROR" +} +``` + +### Response (404 Not Found) +```json +{ + "success": false, + "message": "Data synchronization failed - CKB agency data not found" +} +``` + +--- + +## 3. Trigger API Tool Endpoint Indexing + +### Endpoint +```http +POST /ruuter-public/rag-search/api-tools/index +``` + +Queues an API tool endpoint for vector indexing in Qdrant via CronManager (async). + +### Request Body +```json +{ + "endpointId": "ep-001", + "serviceId": "svc-1", + "name": "Get Pension Status", + "description": "Retrieve the current pension application status for a citizen", + "method": "GET", + "url": "https://api.example.com/pension/status", + "visibility": "public", + "params": [ + { "name": "nationalId", "type": "string", "required": true } + ] +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `endpointId` | string | Yes | Unique endpoint identifier | +| `name` | string | Yes | Endpoint name | +| `description` | string | Yes | Endpoint description | +| `url` | string | Yes | API URL | +| `serviceId` | string | No | Parent service ID | +| `method` | string | No | HTTP method (default: `"GET"`) | +| `visibility` | string | No | `"public"` or `"private"` (default: `"public"`) | +| `type` | string | No | Endpoint type (default: `"custom_endpoint"`) | +| `params` | array | No | List of parameters | + +### Response (200 OK) +```json +{ + "success": true, + "endpoint_id": "ep-001", + "message": "API Tool indexing job queued successfully. Processing asynchronously." +} +``` + +### Response (400 Bad Request) +```json +{ + "success": false, + "error": "MISSING_REQUIRED_FIELDS", + "message": "endpointId, name, description, and url are required" +} +``` + +### Response (500 Internal Server Error) +```json +{ + "success": false, + "error": "INDEXING_QUEUE_FAILED", + "message": "Failed to queue indexing job. CronManager may be unavailable." +} +``` + +--- + +## 4. Enrich and Index Service + +### Endpoint +```http +POST /ruuter-public/rag-search/services/enrich +``` + +Queues a service for enrichment and Qdrant indexing via CronManager (async). + +### Request Body +```json +{ + "service_id": "svc-001", + "name": "Pension Application", + "description": "Submit a new pension application for eligible citizens", + "examples": ["How do I apply for pension?", "Pension eligibility requirements"], + "entities": ["nationalId", "dateOfBirth"], + "ruuter_type": "POST", + "current_state": "active", + "is_common": false +} +``` + +### Request Parameters +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `service_id` | string | Yes | Unique service identifier | +| `name` | string | Yes | Service name | +| `description` | string | Yes | Service description | +| `examples` | array | No | Example user queries | +| `entities` | array | No | Expected entity names | +| `ruuter_type` | string | No | HTTP method (default: `"GET"`) | +| `current_state` | string | No | `"active"`, `"inactive"`, or `"draft"` (default: `"draft"`) | +| `is_common` | boolean | No | Whether this is a common service (default: `false`) | + +### Response (200 OK) +```json +{ + "success": true, + "service_id": "svc-001", + "message": "Service enrichment job queued successfully. Processing asynchronously." +} +``` + +### Response (400 Bad Request) +```json +{ + "success": false, + "error": "MISSING_REQUIRED_FIELDS", + "message": "service_id, name, and description are required" +} +``` + +### Response (500 Internal Server Error) +```json +{ + "success": false, + "error": "ENRICHMENT_QUEUE_FAILED", + "message": "Failed to queue enrichment job. CronManager may be unavailable." +} +``` + +--- \ No newline at end of file diff --git a/docs/API_TOOL_CALLING.md b/docs/API_TOOL_CALLING.md index 5089556b..ce01a35d 100644 --- a/docs/API_TOOL_CALLING.md +++ b/docs/API_TOOL_CALLING.md @@ -12,17 +12,17 @@ loop collects all required parameters from the user before the API call is made. | Component | What it does | Status | |---|---|---| -| **Indexing pipeline** | Takes an endpoint definition → enriches it with LLM context → stores hybrid vectors in Qdrant | ✅ Complete | -| **Tool classifier** | At query time, routes to the best matching endpoint via hybrid search + LLM disambiguation | ✅ Complete | -| **Multi-intent detection** | Score-band gate triggers `IntentDecomposer` (DSPy) to decompose a multi-intent query into focused sub-queries; each sub-query is matched in parallel via `asyncio.gather` | ✅ Phase 1 & 2 Complete | -| **Agentic loop** | Multi-turn parameter collection with session persistence, language-aware clarifying questions, param correction, continuation prompt, and intent-switch detection | ✅ Complete | -| **API caller** | Execute collected params against the real API endpoint, with circuit-breaker protection and localized error handling | ✅ Complete | -| **Response formatter** | Convert raw API JSON into a natural-language answer via DSPy, streamed token-by-token to the GUI | ✅ Complete | -| **Multi-endpoint loop** | Merges param schemas for all parallel endpoints; collects params across turns with a single deduplicated clarifying question per turn; distributes values back per endpoint | ✅ Phase 3 Complete | -| **Parallel API caller** | Fires all completed endpoint calls concurrently via `asyncio.gather` with batch timeout and partial-failure handling | ✅ Phase 4 Complete | -| **Multi-response formatter** | DSPy module that synthesises N API results into a single coherent natural-language answer; supports streaming and blocking execution | ✅ Phase 5 Complete | -| **Full wiring** | `APIToolWorkflowExecutor` routes parallel sessions through `MultiEndpointAgenticLoop` → `MultiAPICaller` → `MultiResponseFormatterModule` with output guardrails | ✅ Phase 6 Complete | -| **ATC Response Cache** | Two-tier Redis cache (L1 exact-match + L2 follow-up context) that eliminates redundant API calls and enables intelligent follow-up handling without re-running the agentic loop | ✅ Complete | +| **Indexing pipeline** | Takes an endpoint definition → enriches it with LLM context → stores hybrid vectors in Qdrant | Complete | +| **Tool classifier** | At query time, routes to the best matching endpoint via hybrid search + LLM disambiguation | Complete | +| **Multi-intent detection** | Score-band gate triggers `IntentDecomposer` (DSPy) to decompose a multi-intent query into focused sub-queries; each sub-query is matched in parallel via `asyncio.gather` | Phase 1 & 2 Complete | +| **Agentic loop** | Multi-turn parameter collection with session persistence, language-aware clarifying questions, param correction, continuation prompt, and intent-switch detection | Complete | +| **API caller** | Execute collected params against the real API endpoint, with circuit-breaker protection and localized error handling | Complete | +| **Response formatter** | Convert raw API JSON into a natural-language answer via DSPy, streamed token-by-token to the GUI | Complete | +| **Multi-endpoint loop** | Merges param schemas for all parallel endpoints; collects params across turns with a single deduplicated clarifying question per turn; distributes values back per endpoint | Phase 3 Complete | +| **Parallel API caller** | Fires all completed endpoint calls concurrently via `asyncio.gather` with batch timeout and partial-failure handling | Phase 4 Complete | +| **Multi-response formatter** | DSPy module that synthesises N API results into a single coherent natural-language answer; supports streaming and blocking execution | Phase 5 Complete | +| **Full wiring** | `APIToolWorkflowExecutor` routes parallel sessions through `MultiEndpointAgenticLoop` → `MultiAPICaller` → `MultiResponseFormatterModule` with output guardrails | Phase 6 Complete | +| **ATC Response Cache** | Two-tier Redis cache (L1 exact-match + L2 follow-up context) that eliminates redundant API calls and enables intelligent follow-up handling without re-running the agentic loop | Complete | --- diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..43a9a532 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,98 @@ +# Architecture + +This page describes how the **LLM Module** fits together, using the +[C4 model](https://c4model.com/) to move from a high-level system view down to the internal +components of the LLM Orchestration Service. Each level links out to the detailed flow documents +that explain the behaviour in depth. + +> The LLM Module is a **multi-workflow orchestrator** for the Bürokratt / Estonian Government AI +> assistant. A tool classifier inspects every user query and routes it to the most appropriate +> workflow — **Service**, **Context**, **RAG**, **API-Tool**, or **Out-of-Domain (OOD)**. +--- + +## Level 1 — System Context + +The context diagram shows the LLM Module as a single system, the people who use it, and the external +systems it depends on (LLM providers, the Central Knowledge Base, observability tooling). + +![LLM Module — C4 System Context Diagram](./images/LLM%20Module%20Context%20Diagram%20(Current).png) + +**Key relationships** + +- **End users / chatbot** send natural-language queries and receive grounded, cited answers. +- **Administrators** configure LLM connections, prompts, budgets, and view analytics. +- **LLM providers** (Azure OpenAI, AWS Bedrock, OpenAI, Anthropic, Google Cloud, self-hosted) supply + chat and embedding models, selected per connection. +- **Central Knowledge Base (CKB)** provides the source content that is indexed for retrieval. +- **Observability** (Langfuse, Grafana/Loki) captures traces, costs, and logs. + +--- + +## Level 2 — Containers + +The container diagram zooms into the deployable units of the system and the data stores they rely on. + +![LLM Module — C4 Container Diagram](./images/LLM%20Module%20App%20Diagram%20(Current).png) + +**Containers & data stores** + +| Container / Store | Role | +| --- | --- | +| **GUI** | Admin web interface for connections, prompts, budgets, and analytics. | +| **Ruuter (public/private)** | API gateway that routes and authorises requests to backend services. | +| **LLM Orchestration Service** | FastAPI service (port `8100`) — the core that runs the workflows. | +| **Notification Server** | Node service pushing real-time updates (e.g. cost alerts, streaming relay). | +| **Qdrant** | Vector database for knowledge-base and API-tool embeddings. | +| **Redis** | Conversation history, session state, and rate-limit counters. | +| **PostgreSQL + ClickHouse** | Relational + columnar stores backing Langfuse analytics. | +| **MinIO (S3)** | Object storage for datasets and documents. | +| **HashiCorp Vault** | Encrypted storage for LLM provider credentials. | + +For how requests traverse the gateway and the orchestration service, see +[API_REFERENCE.md](./API_REFERENCE.md). + +--- + +## Level 3 — Components (LLM Orchestration Service) + +The component diagram opens up the LLM Orchestration Service to show its internal building blocks: the +tool classifier, the per-workflow executors, the contextual retriever, response generation, and +guardrails. + +![LLM Orchestration Service — C4 Component Diagram](./images/LLM%20Orchestration%20Service%20Component%20Diagram%20(Current).png) + +### Request lifecycle (high level) + +1. **Validation & safety** — input is sanitised and checked against guardrails. +2. **Tool classification** — hybrid dense + sparse (BM25) search over indexed examples routes the + query to a workflow. See [HYBRID_SEARCH_CLASSIFICATION.md](./HYBRID_SEARCH_CLASSIFICATION.md). +3. **Workflow execution** — one of: + - **Service** — maps the query to a backend service/intent. + See [TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md](./TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md). + - **Context** — greeting/conversation handling with Redis-backed history. + See [CONTEXT_WORKFLOW_GREETING_DETECTION.md](./CONTEXT_WORKFLOW_GREETING_DETECTION.md). + - **RAG** — contextual retrieval (hybrid search + RRF fusion) then grounded generation. + See [CONTEXTUAL_RETRIEVAL_FLOW.md](./CONTEXTUAL_RETRIEVAL_FLOW.md). + - **API-Tool** — agentic multi-endpoint API calling. + See [API_TOOL_CALLING.md](./API_TOOL_CALLING.md). + - **OOD** — graceful fallback for out-of-domain queries. +4. **Generation & guardrails** — the response is generated with citations and re-checked before return. +5. **Observability** — the full trace and cost are recorded in Langfuse; logs go to Loki. + +### Component → documentation map + +| Concern | Detailed doc | +| --- | --- | +| Tool classifier overview | [TOOL_CLASSIFIER.md](./TOOL_CLASSIFIER.md) | +| Hybrid search & intent enrichment | [HYBRID_SEARCH_CLASSIFICATION.md](./HYBRID_SEARCH_CLASSIFICATION.md) | +| Contextual retrieval (RAG) | [CONTEXTUAL_RETRIEVAL_FLOW.md](./CONTEXTUAL_RETRIEVAL_FLOW.md) | +| API tool calling | [API_TOOL_CALLING.md](./API_TOOL_CALLING.md) | +| Service workflow (UI trace) | [TESTPRODUCTIONLLM_SERVICE_WORKFLOW.md](./TESTPRODUCTIONLLM_SERVICE_WORKFLOW.md) | +| Conversation history & sessions | [REDIS_SESSION_STORE.md](./REDIS_SESSION_STORE.md), [CONTEXT_WORKFLOW_GREETING_DETECTION.md](./CONTEXT_WORKFLOW_GREETING_DETECTION.md) | +| LLM credentials & Vault | [LLM_CONFIG_VAULT_INTEGRATION.md](./LLM_CONFIG_VAULT_INTEGRATION.md), [VAULT_SETUP_AND_USAGE.md](./VAULT_SETUP_AND_USAGE.md), [VAULT_SECURITY_ARCHITECTURE.md](./VAULT_SECURITY_ARCHITECTURE.md) | +| Connection swapping | [CONNECTION_SWAP_FLOW.md](./CONNECTION_SWAP_FLOW.md) | +| Prompt configuration | [CUSTOM_PROMPT_CONFIGURATION.md](./CUSTOM_PROMPT_CONFIGURATION.md) | + +--- + +For the full catalogue of documentation, see the [Documentation Index](./README.md). \ No newline at end of file diff --git a/docs/CONTEXTUAL_RETRIEVAL_FLOW.md b/docs/CONTEXTUAL_RETRIEVAL_FLOW.md index c59c342c..9c6c0d5b 100644 --- a/docs/CONTEXTUAL_RETRIEVAL_FLOW.md +++ b/docs/CONTEXTUAL_RETRIEVAL_FLOW.md @@ -82,12 +82,12 @@ For each of the 6 refined queries, the system performs parallel semantic and BM2 - **<0.3**: Likely irrelevant **0.4 is the optimal balance** because: -- ✅ Captures semantically related content beyond exact matches -- ✅ Includes contextual information (e.g., implementation details, legal context) -- ✅ Maintains quality while maximizing diversity -- ✅ Industry standard for production RAG systems -- ❌ Lower values (0.3) introduce too much noise -- ❌ Higher values (0.5+) miss valuable context +- Captures semantically related content beyond exact matches +- Includes contextual information (e.g., implementation details, legal context) +- Maintains quality while maximizing diversity +- Industry standard for production RAG systems +- Lower values (0.3) introduce too much noise +- Higher values (0.5+) miss valuable context **Performance Impact:** - Threshold 0.5: ~17 results, 4 unique chunks (too narrow) @@ -172,10 +172,10 @@ The k-parameter determines how quickly scores decay with rank position: | k=90 | 0.0110 | 0.0100 | Very narrow | Too democratic | **k=35 Advantages:** -- ✅ **65-70% higher top-rank scores** vs k=60 (0.0541 vs 0.0328) -- ✅ **Clear score separation** between highly relevant and marginal chunks -- ✅ **Balanced approach** - respects both top results and broader context -- ✅ **Better signal for response generator** - easier to identify best chunks +- **65-70% higher top-rank scores** vs k=60 (0.0541 vs 0.0328) +- **Clear score separation** between highly relevant and marginal chunks +- **Balanced approach** - respects both top results and broader context +- **Better signal for response generator** - easier to identify best chunks **Score Differentiation Example:** ``` @@ -302,12 +302,12 @@ For each of the top 10 chunks: | Metric | Value | Target | Status | |--------|-------|--------|--------| -| Semantic Results per Query | 27.3 | >5 | ✅ Excellent | -| Unique Semantic Chunks | 42 | >10 | ✅ Excellent | -| Fusion Coverage | 100% | >80% | ✅ Perfect | -| Both-sources Validation | 12/12 | >50% | ✅ Perfect | -| Score Differentiation | High | Clear gaps | ✅ Excellent | -| Retrieval Speed | 1.6s | <3s | ✅ Excellent | +| Semantic Results per Query | 27.3 | >5 | Excellent | +| Unique Semantic Chunks | 42 | >10 | Excellent | +| Fusion Coverage | 100% | >80% | Perfect | +| Both-sources Validation | 12/12 | >50% | Perfect | +| Score Differentiation | High | Clear gaps | Excellent | +| Retrieval Speed | 1.6s | <3s | Excellent | --- @@ -573,10 +573,10 @@ When evaluating the quality of the contextual retrieval system and response gene ### Alert Thresholds -- ⚠️ Semantic yield drops below 5 results/query -- ⚠️ Fusion coverage drops below 80% -- ⚠️ Retrieval time exceeds 3 seconds -- ⚠️ BM25 index build fails or incomplete +- Semantic yield drops below 5 results/query +- Fusion coverage drops below 80% +- Retrieval time exceeds 3 seconds +- BM25 index build fails or incomplete --- diff --git a/docs/CUSTOM_PROMPT_CONFIGURATION.md b/docs/CUSTOM_PROMPT_CONFIGURATION.md index 8a7f94ef..4647a26c 100644 --- a/docs/CUSTOM_PROMPT_CONFIGURATION.md +++ b/docs/CUSTOM_PROMPT_CONFIGURATION.md @@ -2,14 +2,58 @@ ## Overview -The custom prompt configuration system allows admins to configure prompts via UI that automatically apply to all response generation operations. Changes are cached with a 5-minute TTL and can be immediately refreshed when updated. +The custom prompt configuration system allows admins to configure a single organisation-level prompt via the UI that automatically applies to user-facing answer generation. Changes are cached with a 5-minute TTL and can be immediately refreshed when updated. + +The same configured prompt is consumed by **two** workflows — the **RAG workflow** and the **API Tool Calling workflow** — through one shared `PromptConfigurationLoader`. The **Context workflow** does **not** apply custom prompts (greetings use static templates and history answers use their own signature). See [Where Custom Prompts Are Applied (by Workflow)](#where-custom-prompts-are-applied-by-workflow). + +--- + +## Where Custom Prompts Are Applied (by Workflow) + +All consumers read the same prompt from the shared `PromptConfigurationLoader` +(`src/utils/prompt_config_loader.py`, 5-minute TTL cache). They differ in **how** they inject it. + +| Workflow | Custom prompt applied? | Where / how | +|---|---|---| +| **RAG** | Yes | `ResponseGeneratorAgent` — the prompt is wrapped as `[SYSTEM INSTRUCTIONS]…[USER QUESTION]` and appended to the question for both streaming and non-streaming generation. | +| **API Tool Calling** | Yes | `APIToolWorkflowExecutor._get_custom_instructions()` loads the raw prompt and passes it into parameter extraction and response formatting (see below). | +| **Context** | No | `context_workflow.py` / `context_analyzer.py` do not load or apply the custom prompt. Greetings return static templates; history answers use `ContextResponseGenerationSignature` without injection. | +| **Service** | No | The response is pre-formed text from Ruuter/DMapper — there is no LLM generation step to steer. | +| **OOD** | No | Fixed localized out-of-scope message. | + +### RAG workflow + +- Source: [`src/llm_orchestration_service.py`](../src/llm_orchestration_service.py) → `_get_custom_instructions_for_response_generation()` builds the prefix + `"[SYSTEM INSTRUCTIONS]\n{prompt}\n\n[USER QUESTION]\n"` and passes it as + `ResponseGeneratorAgent(custom_instructions_prefix=…)`. +- Application: [`src/response_generator/response_generate.py`](../src/response_generator/response_generate.py) applies it as + `augmented_question = f"{question}\n\n{custom_instructions_prefix}"` in both `forward()` and + `stream_response()`. +- **Note:** despite the name `custom_instructions_prefix`, the string is **appended after** the + question (the wrapper text itself carries the `[USER QUESTION]` marker). It is **not** applied to + `PromptRefinerAgent`, which only optimises the query for retrieval. + +### API Tool Calling workflow + +- Source: [`src/tool_classifier/workflows/api_tool_workflow.py`](../src/tool_classifier/workflows/api_tool_workflow.py) → `_get_custom_instructions()` reads the **same** `prompt_config_loader` + (via `asyncio.to_thread`, fail-open to `""`). Unlike the RAG path, it passes the **raw** prompt + (no `[SYSTEM INSTRUCTIONS]` wrapper). +- It is injected as a dedicated DSPy `custom_instructions` input field into: + - `ParamExtractionModule` ([`param_extractor.py`](../src/tool_classifier/param_extractor.py)) — steers how parameters are extracted from the user. + - `APIResponseFormatterModule` ([`api_response_formatter.py`](../src/tool_classifier/api_response_formatter.py)) — single-endpoint natural-language answer. + - `MultiResponseFormatterModule` ([`multi_response_formatter.py`](../src/tool_classifier/multi_response_formatter.py)) — multi-endpoint synthesis. +- In the formatter/extractor signatures, a non-empty `custom_instructions` is followed with + **HIGHEST PRIORITY**, overriding defaults such as language policy, tone, and formatting. +- Additionally, the workflow derives the response language from the prompt via + `_language_from_custom_instructions()` and merges any Redis conversation summary into the same + `custom_instructions` string before extraction. --- ## Architecture Components ### 1. **Database Layer** -- **Table**: `public.prompt_configuration` +- **Table**: `rag_search.prompt_configuration` - **Columns**: `id` (BIGINT), `prompt` (TEXT) - Stores the custom prompt text configured by admins @@ -36,7 +80,7 @@ The custom prompt configuration system allows admins to configure prompts via UI - **ResponseGeneratorAgent** (`src/response_generator/response_generate.py`) - Accepts `custom_instructions_prefix` parameter - - Prepends custom instructions to user questions + - Appends custom instructions after the user question - Applied in both streaming and non-streaming modes ### 4. **API Endpoints** @@ -120,8 +164,8 @@ The custom prompt configuration system allows admins to configure prompts via UI ▼ ┌─────────────────────────────────────────────────────────────────┐ │ ResponseGeneratorAgent.forward() or stream_response() │ -│ - Prepends custom_instructions_prefix to user question │ -│ - Modified question = "{prefix}{user_question}" │ +│ - Appends custom_instructions_prefix after user question │ +│ - Modified question = "{user_question}{prefix}" │ └────────────────┬────────────────────────────────────────────────┘ │ ▼ @@ -169,8 +213,7 @@ The custom prompt configuration system allows admins to configure prompts via UI ### **User Request Processing** 1. **Request Received** (Any of 3 endpoints) - - `/orchestrate` - Standard response - - `/orchestrate/test` - Test response + - `/orchestrate/test` - Test Sresponse - `/orchestrate/stream` - Streaming response 2. **Service Components Initialization** @@ -242,7 +285,7 @@ RAG_SEARCH_PROMPT_REFRESH=http://llm-orchestration-service:8100/prompt-config/re ### **1. Insert Test Prompt** ```sql -INSERT INTO public.prompt_configuration (id, prompt) +INSERT INTO rag_search.prompt_configuration (id, prompt) VALUES (1, 'Always respond in Estonian language. Be professional and concise.') ON CONFLICT (id) DO UPDATE SET prompt = EXCLUDED.prompt; ``` @@ -260,7 +303,7 @@ curl -X POST http://localhost:8100/orchestrate/test \ ### **3. Update Prompt** ```sql -UPDATE public.prompt_configuration +UPDATE rag_search.prompt_configuration SET prompt = 'Provide concise answers using bullet points. Be helpful and clear.' WHERE id = 1; ``` @@ -290,14 +333,14 @@ curl -X POST http://localhost:8100/prompt-config/refresh ## Key Features -✅ **TTL Caching** - 5-minute cache reduces database calls -✅ **Immediate Updates** - Admin changes trigger instant refresh -✅ **Graceful Degradation** - If refresh fails, TTL cache continues working -✅ **Thread-Safe** - Multiple concurrent requests handled safely -✅ **Retry Logic** - 3 attempts with exponential backoff for HTTP failures -✅ **Instruction Prepending** - Preserves DSPy optimization compatibility -✅ **Applied Consistently** - Works across all 3 orchestration endpoints -✅ **Applied to ResponseGenerator Only** - Not applied to PromptRefinerAgent + **TTL Caching** - 5-minute cache reduces database calls + **Immediate Updates** - Admin changes trigger instant refresh + **Graceful Degradation** - If refresh fails, TTL cache continues working + **Thread-Safe** - Multiple concurrent requests handled safely + **Retry Logic** - 3 attempts with exponential backoff for HTTP failures + **Instruction Appending** - Custom instructions appended to the question without modifying the DSPy signature + **Applied Consistently** - Works across all 3 orchestration endpoints + **Applied to RAG & API Tool Calling** - In RAG, only the ResponseGenerator (not the PromptRefiner); in API Tool Calling, the param extractor and response formatters. Not applied to the Context workflow. --- @@ -322,10 +365,10 @@ Context: [retrieved documentation chunks...] ``` **Expected Response:** -- In Estonian language ✅ -- Professional tone ✅ -- Concise format ✅ -- Citations included ✅ +- In Estonian language +- Professional tone +- Concise format +- Citations included --- @@ -348,7 +391,7 @@ Context: [retrieved documentation chunks...] ### **Prompt Not Applied** - Check logs for: "Custom prompt configuration loaded at startup" -- Verify database has prompt: `SELECT * FROM public.prompt_configuration;` +- Verify database has prompt: `SELECT * FROM rag_search.prompt_configuration;` - Test refresh endpoint: `curl -X POST http://localhost:8100/prompt-config/refresh` ### **Cache Not Refreshing** @@ -365,7 +408,9 @@ Context: [retrieved documentation chunks...] ## Notes -- Custom prompts apply **only to ResponseGeneratorAgent** (not PromptRefinerAgent) -- PromptRefiner focuses on query optimization for retrieval -- ResponseGenerator needs language policy and interaction style for user-facing content -- This design preserves DSPy optimization compatibility by using instruction prepending instead of signature modification +- In the **RAG** path, custom prompts apply **only to `ResponseGeneratorAgent`** (not `PromptRefinerAgent`). + The wrapped instructions are appended to the question rather than modifying the DSPy signature. +- The **API Tool Calling** workflow consumes the **same** configured prompt (via the shared loader) and + injects the raw text as a dedicated `custom_instructions` DSPy input field on the parameter extractor + and the response formatters, where it is followed with highest priority. +- The **Context** and **Service** workflows do not apply custom prompts (see the per-workflow section). diff --git a/docs/LLM_CONFIG_VAULT_INTEGRATION.md b/docs/LLM_CONFIG_VAULT_INTEGRATION.md deleted file mode 100644 index 563054ee..00000000 --- a/docs/LLM_CONFIG_VAULT_INTEGRATION.md +++ /dev/null @@ -1,546 +0,0 @@ -# LLM Config Module - HashiCorp Vault Integration - -## Overview - -The LLM Config Module integrates with HashiCorp Vault to securely store and manage API keys, endpoints, and other sensitive configuration data for various LLM providers (AWS Bedrock, Azure OpenAI, etc.). This integration replaces the traditional `.env` file approach with a more secure, centralized secret management system. - -## Architecture - -### Components - -1. **VaultSecretResolver** - Core component that interfaces with Vault -2. **ConfigurationLoader** - Loads configuration and resolves secrets from Vault -3. **LLMManager** - Main entry point that initializes with Vault-backed configuration -4. **Connection Management** - Dynamic discovery of provider connections from Vault - -### Key Features - -- **Environment-Aware**: Automatically discovers and uses appropriate secrets based on environment (production/development/test) -- **User-Independent**: No hardcoded user lists - dynamically discovers available connections -- **Provider Discovery**: Automatically detects which LLM providers are available based on Vault contents -- **Fallback Protection**: Graceful handling when Vault is unavailable (fails securely) - -## Vault Data Structure - -### Secret Storage Schema - -The Vault integration uses the KV v2 secrets engine with the following hierarchical structure: - -``` -secret/ -├── users/ -│ ├── user1/ -│ │ ├── conn_12345abc/ -│ │ │ ├── data/ -│ │ │ │ ├── provider: "aws_bedrock" -│ │ │ │ ├── environment: "production" -│ │ │ │ ├── aws_access_key_id: "AKIA..." -│ │ │ │ ├── aws_secret_access_key: "..." -│ │ │ │ ├── aws_region: "us-east-1" -│ │ │ │ └── model_id: "anthropic.claude-3-sonnet-20240229-v1:0" -│ │ └── conn_67890def/ -│ │ ├── data/ -│ │ │ ├── provider: "azure_openai" -│ │ │ ├── environment: "development" -│ │ │ ├── api_key: "sk-..." -│ │ │ ├── endpoint: "https://myservice.openai.azure.com/" -│ │ │ ├── deployment_name: "gpt-4" -│ │ │ └── api_version: "2024-02-15-preview" -│ └── user2/ -│ └── conn_11111xyz/ -│ └── data/ -│ ├── provider: "aws_bedrock" -│ ├── environment: "production" -│ └── ... -``` - -### Connection Metadata - -Each connection contains: - -- **Provider Type**: `aws_bedrock`, `azure_openai`, etc. -- **Environment**: `production`, `development`, `test` -- **Provider-specific secrets**: API keys, endpoints, regions, model IDs -- **Connection ID**: Unique identifier for the connection - -## Development Container Setup - -### Current Container Configuration - -The project includes a development Vault container configured in `docker-compose.yml`: - -```yaml -vault: - image: hashicorp/vault:latest - container_name: vault - command: ["vault", "server", "-dev", "-dev-listen-address=0.0.0.0:8200", "-dev-root-token-id=myroot"] - cap_add: - - IPC_LOCK - ports: - - "8200:8200" - environment: - - VAULT_ADDR=http://0.0.0.0:8200 - - VAULT_API_ADDR=http://localhost:8200 - - VAULT_DEV_ROOT_TOKEN_ID=myroot - - VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 - volumes: - - vault-data:/vault/data - networks: - - bykstack - restart: unless-stopped - healthcheck: - test: ["CMD", "vault", "status"] - interval: 10s - timeout: 5s - retries: 5 -``` - -### Starting the Development Environment - -1. **Start Vault Container**: - ```bash - docker-compose up vault -d - ``` - -2. **Verify Vault is Running**: - ```bash - curl http://localhost:8200/v1/sys/health - ``` - -3. **Access Vault UI**: - - URL: http://localhost:8200 - - Token: `myroot` - -### Development Configuration - -For development, set these environment variables: - -```bash -export VAULT_ADDR="http://localhost:8200" -export VAULT_TOKEN="myroot" -``` - -## Usage Examples - -### Production Environment - -```python -import os -from llm_config_module import LLMManager - -# Set Vault connection details -os.environ["VAULT_ADDR"] = "https://vault.company.com" -os.environ["VAULT_TOKEN"] = "your-production-token" - -# Initialize LLM Manager - automatically discovers production providers -manager = LLMManager(environment="production") - -# Get available providers (discovered from Vault) -providers = manager.get_available_providers() -print(f"Available providers: {list(providers.keys())}") - -# Use the LLM -llm = manager.get_llm() -response = llm.generate("Hello, world!") -``` - -### Development Environment - -```python -# Development requires a specific connection ID -manager = LLMManager( - environment="development", - connection_id="conn_12345abc" # Specific dev connection -) - -llm = manager.get_llm() -``` - -### Dynamic Provider Discovery - -The system automatically discovers which providers are available: - -```python -manager = LLMManager(environment="production") - -# Only providers with valid Vault secrets will be available -if manager.is_provider_available(LLMProvider.AWS_BEDROCK): - print("AWS Bedrock is configured and available") - -if manager.is_provider_available(LLMProvider.AZURE_OPENAI): - print("Azure OpenAI is configured and available") -``` - -## Configuration Details - -### Vault Configuration (llm_config.yaml) - -```yaml -vault: - enabled: true - url: "${VAULT_ADDR}" - token: "${VAULT_TOKEN}" - mount_point: "secret" - secrets_engine: "kv-v2" - -providers: - aws_bedrock: - enabled: true # Will be dynamically determined from Vault - model_id: "anthropic.claude-3-sonnet-20240229-v1:0" - max_tokens: 1000 - temperature: 0.7 - - azure_openai: - enabled: true # Will be dynamically determined from Vault - max_tokens: 1000 - temperature: 0.7 -``` - -### Environment Variable Resolution - -The configuration supports environment variable substitution: - -- `${VAULT_ADDR}` - Vault server URL -- `${VAULT_TOKEN}` - Vault authentication token - -## Production Considerations - -### Security Best Practices - -#### 1. Authentication & Authorization - -**🔒 Token Management**: -```bash -# Use short-lived tokens in production -vault write auth/userpass/users/llm-service password="secure-password" policies="llm-read-policy" - -# Generate service token -vault write -field=token auth/userpass/login/llm-service password="secure-password" -``` - -**🔒 Policy Configuration**: -```hcl -# llm-read-policy.hcl -path "secret/data/users/*/conn_*" { - capabilities = ["read"] -} - -path "secret/metadata/users/*" { - capabilities = ["list", "read"] -} -``` - -#### 2. Network Security - -**🔒 TLS Configuration**: -```hcl -# vault.hcl (Production) -listener "tcp" { - address = "0.0.0.0:8200" - tls_cert_file = "/etc/ssl/vault/vault.crt" - tls_key_file = "/etc/ssl/vault/vault.key" - tls_min_version = "tls12" -} -``` - -**🔒 Network Isolation**: -- Deploy Vault in private subnets -- Use VPC endpoints for AWS services -- Implement network ACLs and security groups -- Enable Vault audit logging - -#### 3. High Availability Setup - -**🏗️ Raft Storage Backend**: -```hcl -storage "raft" { - path = "/vault/data" - node_id = "vault-1" - - retry_join { - leader_api_addr = "https://vault-1.internal:8200" - } - retry_join { - leader_api_addr = "https://vault-2.internal:8200" - } - retry_join { - leader_api_addr = "https://vault-3.internal:8200" - } -} -``` - -**🏗️ Auto-Unseal** (recommended): -```hcl -seal "awskms" { - region = "us-east-1" - kms_key_id = "alias/vault-unseal-key" -} -``` - -#### 4. Monitoring & Logging - -**📊 Health Checks**: -```yaml -# kubernetes health check -livenessProbe: - httpGet: - path: /v1/sys/health - port: 8200 - scheme: HTTPS - initialDelaySeconds: 60 - timeoutSeconds: 5 -``` - -**📊 Audit Logging**: -```hcl -audit "file" { - file_path = "/vault/logs/audit.log" -} -``` - -#### 5. Backup & Recovery - -**💾 Automated Snapshots**: -```bash -#!/bin/bash -# backup-vault.sh -vault operator raft snapshot save "vault-snapshot-$(date +%Y%m%d-%H%M%S).snap" -aws s3 cp "vault-snapshot-*.snap" s3://vault-backups/ -``` - -### Production Deployment Architecture - -```mermaid -graph TB - subgraph "Load Balancer" - ALB[Application Load Balancer] - end - - subgraph "Vault Cluster" - V1[Vault Node 1
Active] - V2[Vault Node 2
Standby] - V3[Vault Node 3
Standby] - end - - subgraph "Application Tier" - APP1[LLM App 1] - APP2[LLM App 2] - APP3[LLM App 3] - end - - subgraph "External Services" - AWS[AWS Bedrock] - AZURE[Azure OpenAI] - end - - ALB --> V1 - ALB --> V2 - ALB --> V3 - - APP1 --> ALB - APP2 --> ALB - APP3 --> ALB - - APP1 --> AWS - APP2 --> AZURE - APP3 --> AWS -``` - -### Environment-Specific Configurations - -#### Production -```yaml -# Production values -vault: - url: "https://vault.company.com" - token: "${VAULT_SERVICE_TOKEN}" # From secure secret management - -# Use IAM roles where possible -providers: - aws_bedrock: - use_iam_role: true # Preferred over access keys -``` - -#### Staging -```yaml -vault: - url: "https://vault-staging.company.com" - token: "${VAULT_STAGING_TOKEN}" -``` - -#### Development -```yaml -vault: - url: "http://localhost:8200" - token: "myroot" # Development only -``` - -## Migration from .env Files - -### Step-by-Step Migration - -1. **Identify Current Secrets**: - ```bash - # List current .env variables - grep -E "(API_KEY|SECRET|TOKEN)" .env - ``` - -2. **Create Vault Connections**: - ```bash - # Example: Migrate AWS credentials - vault kv put secret/users/production/conn_aws_prod \ - provider="aws_bedrock" \ - environment="production" \ - aws_access_key_id="$AWS_ACCESS_KEY_ID" \ - aws_secret_access_key="$AWS_SECRET_ACCESS_KEY" \ - aws_region="us-east-1" \ - model_id="anthropic.claude-3-sonnet-20240229-v1:0" - ``` - -3. **Update Application Code**: - ```python - # Before (using .env) - manager = LLMManager(config_path="config.yaml", environment="production") - - # After (using Vault) - manager = LLMManager(environment="production") # Auto-discovers from Vault - ``` - -4. **Verify Migration**: - ```python - # Test that providers are discovered correctly - providers = manager.get_available_providers() - assert len(providers) > 0, "No providers discovered from Vault" - ``` - -## Testing - -### Unit Tests - -The integration includes comprehensive test coverage: - -- **Vault Integration Tests**: `test_integration_vault_llm_config.py` -- **Provider-Specific Tests**: `test_aws.py`, `test_azure.py` -- **Helper Functions**: `vault_test_helpers.py` - -### Running Tests - -```bash -# Run all tests -uv run pytest -v - -# Run only Vault integration tests -uv run pytest tests/test_integration_vault_llm_config.py -v - -# Run provider-specific tests -uv run pytest tests/test_aws.py tests/test_azure.py -v -``` - -### Test Helpers - -The `vault_test_helpers.py` provides utilities for test discovery: - -```python -from tests.vault_test_helpers import ( - check_vault_available, - get_available_providers_from_vault, - should_skip_aws_test, - should_skip_azure_test -) - -# Conditionally skip tests based on Vault provider availability -@pytest.mark.skipif(should_skip_aws_test(), reason="AWS not available in Vault") -def test_aws_integration(): - # Test will only run if AWS Bedrock is configured in Vault - pass -``` - -## Troubleshooting - -### Common Issues - -#### 1. Vault Connection Failures -```python -# Check Vault connectivity -try: - from rag_config_manager.vault import VaultClient - vault = VaultClient() - print(f"Vault available: {vault.is_vault_available()}") -except Exception as e: - print(f"Vault error: {e}") -``` - -#### 2. Provider Discovery Issues -```python -# Debug provider discovery -import os -os.environ["VAULT_ADDR"] = "http://localhost:8200" -os.environ["VAULT_TOKEN"] = "myroot" - -manager = LLMManager(environment="production") -providers = manager.get_available_providers() -print(f"Discovered providers: {list(providers.keys())}") -``` - -#### 3. Authentication Errors -- Verify `VAULT_TOKEN` is valid and not expired -- Check token policies have required permissions -- Ensure Vault server is accessible from application network - -#### 4. Secret Path Issues -- Verify secret paths match the expected structure -- Check that secrets exist in the correct mount point -- Ensure proper KV v2 format is used - -### Logging - -Enable debug logging to troubleshoot issues: - -```python -import logging -logging.basicConfig(level=logging.DEBUG) - -# The LLM Config Module uses loguru for logging -from loguru import logger -logger.add("vault_debug.log", level="DEBUG") -``` - -## Best Practices Summary - -### ✅ Do: -- Use production-grade Vault deployment with HA -- Implement proper authentication (avoid root tokens) -- Enable TLS in production -- Use auto-unseal mechanisms -- Implement comprehensive monitoring -- Regular backup and recovery testing -- Use IAM roles where possible instead of static keys -- Rotate secrets regularly - -### ❌ Don't: -- Use development mode Vault in production -- Store root tokens in application code -- Disable TLS in production environments -- Skip audit logging -- Use overly permissive policies -- Store Vault tokens in environment files -- Forget to implement proper secret rotation - -## Support & Maintenance - -### Vault Version Compatibility -- **Minimum**: Vault 1.12+ -- **Recommended**: Vault 1.15+ -- **Tested With**: Vault 1.15.1 - -### Dependencies -- `rag_config_manager` - Vault client interface -- `hvac` - HashiCorp Vault client library -- `pydantic` - Data validation and settings management - -### Monitoring Endpoints -- Health: `GET /v1/sys/health` -- Metrics: `GET /v1/sys/metrics` (Prometheus format) -- Status: `vault status` (CLI command) - -This integration provides a robust, secure, and scalable approach to managing LLM provider secrets using HashiCorp Vault, replacing traditional environment variable-based configuration with enterprise-grade secret management. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..32115576 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,64 @@ +# Documentation Index + +Welcome to the documentation for the **LLM Module** — the LLM orchestration component of the +Bürokratt / Estonian Government AI assistant. This index catalogues every document under `docs/`, +grouped by topic. Start with the [project README](../README.md) for the big picture, then dive into +the areas below. + +> New to the system? Read [ARCHITECTURE.md](./ARCHITECTURE.md) first — it walks the C4 diagrams from +> system context down to the internal components. + +--- + +## Architecture & Design + +| Document | What it covers | +| --- | --- | +| [ARCHITECTURE.md](./ARCHITECTURE.md) | C4 model walkthrough (context → containers → components) with links into every detailed flow. The recommended starting point. | + +## Retrieval & Search + +| Document | What it covers | +| --- | --- | +| [CONTEXTUAL_RETRIEVAL_FLOW.md](./CONTEXTUAL_RETRIEVAL_FLOW.md) | The RAG workflow in depth: multi-query expansion, hybrid (semantic + BM25) search, RRF rank fusion, thresholds, and quality testing. | +| [HYBRID_SEARCH_CLASSIFICATION.md](./HYBRID_SEARCH_CLASSIFICATION.md) | Tool-classifier architecture using per-example dense (3072-dim) + sparse (BM25) vectors in Qdrant; offline indexing and query-time classification. | + +## Tool Classification & Workflows + +| Document | What it covers | +| --- | --- | +| [TOOL_CLASSIFIER.md](./TOOL_CLASSIFIER.md) | High-level overview of the classifier: routing model, key functions, and configuration. Start here, then read the per-workflow docs below. | +| [TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md](./TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md) | Service-workflow architecture: high-confidence vs ambiguous routes and service-discovery logic. | +| [CONTEXT_WORKFLOW_GREETING_DETECTION.md](./CONTEXT_WORKFLOW_GREETING_DETECTION.md) | The Context workflow: greeting detection and Redis-backed conversation history with incremental summaries. | +| [API_TOOL_CALLING.md](./API_TOOL_CALLING.md) | The agentic API-Tool workflow end to end: indexing, multi-intent decomposition, multi-endpoint agentic loop, calling, and response formatting. | +| [TESTPRODUCTIONLLM_SERVICE_WORKFLOW.md](./TESTPRODUCTIONLLM_SERVICE_WORKFLOW.md) | Service workflow + streaming via the TestProductionLLM page (three-hop SSE relay). | + +## Configuration & Secrets + +| Document | What it covers | +| --- | --- | +| [LLM_CONFIG_VAULT_INTEGRATION.md](./LLM_CONFIG_VAULT_INTEGRATION.md) | HashiCorp Vault integration for LLM credentials: KV v2 layout, dev setup, production HA, and migration from `.env`. | +| [VAULT_SETUP_AND_USAGE.md](./VAULT_SETUP_AND_USAGE.md) | Operational guide: dual-network topology, vault-agents, bootstrap flow, AppRole auth, and credential reconciliation. | +| [VAULT_SECURITY_ARCHITECTURE.md](./VAULT_SECURITY_ARCHITECTURE.md) | Security model: threat model, network isolation, AppRole authentication, and the per-policy access-control matrix. | +| [CONNECTION_SWAP_FLOW.md](./CONNECTION_SWAP_FLOW.md) | UUID-based Vault path design enabling zero-I/O environment swaps (promote/demote) between LLM connections. | +| [CUSTOM_PROMPT_CONFIGURATION.md](./CUSTOM_PROMPT_CONFIGURATION.md) | Admin-facing prompt management: database → Ruuter DSL → Python loader with TTL cache and invalidation. | + +## Data & Sessions + +| Document | What it covers | +| --- | --- | +| [REDIS_SESSION_STORE.md](./REDIS_SESSION_STORE.md) | Redis session-store usage: CRUD for agentic-loop state, TTL behaviour, and the async API. | + +## API Reference + +| Document | What it covers | +| --- | --- | +| [API_REFERENCE.md](./API_REFERENCE.md) | HTTP API reference: LLM Connections management, Inference Results storage/retrieval, and the chatbot Inquiry endpoint. | + +--- + +## Related resources + +- [Project README](../README.md) — overview, quick start, and component map. +- [CONTRIBUTING.md](../CONTRIBUTING.md) — development environment, tooling, and CI checks. +- Architecture diagrams (source images): [`images/`](./images). diff --git a/docs/TESTMODEL_SERVICE_WORKFLOW.md b/docs/TESTMODEL_SERVICE_WORKFLOW.md deleted file mode 100644 index e1649a9f..00000000 --- a/docs/TESTMODEL_SERVICE_WORKFLOW.md +++ /dev/null @@ -1,405 +0,0 @@ -# TestModel Page — Service Workflow Documentation - -This document traces the **service workflow** end-to-end through the **TestModel** UI page. It covers two scenarios: - -1. **Natural-language service detection** — the user types a free-text query that the system classifies as a service. -2. **MCQ button-click** — the user clicks a choice button whose payload is a `#service` command, short-circuiting the NLU pipeline. - ---- - -## Architecture Overview - -``` -┌─────────────┐ POST /rag-search/inference/test ┌─────────────────┐ -│ TestModel │ ──────────────────────────────────────────▷ │ Ruuter (proxy) │ -│ (GUI) │ │ /rag-search/ │ -│ index.tsx │ ◁─────────── JSON response ──────────────── │ inference/test │ -└─────────────┘ └───────┬─────────┘ - │ - POST /orchestrate/test - ▼ - ┌───────────────────────┐ - │ llm_orchestration_ │ - │ service_api.py │ - │ test_orchestrate_ │ - │ llm_request() │ - └───────┬───────────────┘ - │ - OrchestrationRequest (mapped with defaults) - ▼ - ┌───────────────────────┐ - │ llm_orchestration_ │ - │ service.py │ - │ process_orchestration_ │ - │ request() │ - └───────────────────────┘ -``` - ---- - -## Key Files - -| Layer | File | Purpose | -|---|---|---| -| **GUI** | `GUI/src/pages/TestModel/index.tsx` | UI page with connection selector, text input, result display, MCQ buttons | -| **GUI Service** | `GUI/src/services/inference.ts` | `viewInferenceResult()` — POST to `/rag-search/inference/test` | -| **API Layer** | `src/llm_orchestration_service_api.py` | `/orchestrate/test` handler — maps `TestOrchestrationRequest` → `OrchestrationRequest` and calls `process_orchestration_request()` | -| **Orchestration** | `src/llm_orchestration_service.py` | `process_orchestration_request()` — the core pipeline (language detection → `#service` prefix check → query validation → guardrails → classifier → service workflow) | -| **Service Workflow** | `src/tool_classifier/workflows/service_workflow.py` | `ServiceWorkflowExecutor` — service discovery, intent detection, entity extraction, endpoint call, direct step execution | -| **Models** | `src/models/request_models.py` | `OrchestrationRequest`, `OrchestrationResponse`, `TestOrchestrationRequest`, `TestOrchestrationResponse`, `ChoiceButton` | -| **Constants** | `src/tool_classifier/constants.py` | `SERVICE_STEP_PREFIXES`, `RUUTER_SERVICE_BASE_URL`, search thresholds | - ---- - -## Flow 1: Natural-Language Service Detection - -### 1.1 Frontend — User Sends a Message - -The user selects an LLM connection from the dropdown and types a query (e.g., *"My keyboard is not working"*). - -**`TestModel/index.tsx` → `handleSend()`** (line 72): -```tsx -inferenceMutation.mutate({ - llmConnectionId: Number(testLLM.connectionId), - message: testLLM.text, -}); -``` - -**`inference.ts` → `viewInferenceResult()`** (line 50): -```ts -const { data } = await apiDev.post(inferenceEndpoints.VIEW_TEST_INFERENCE_RESULT(), { - connectionId: request.llmConnectionId, - message: request.message, -}); -``` - -This POST goes to `/rag-search/inference/test` (via Ruuter proxy), which maps to the backend `/orchestrate/test` endpoint. - -### 1.2 API Layer — Request Mapping - -**`llm_orchestration_service_api.py` → `test_orchestrate_llm_request()`** (line 313): -- Receives a `TestOrchestrationRequest` (only `message`, `environment`, optional `connectionId`). -- Maps to a full `OrchestrationRequest` with defaults: - -```python -full_request = OrchestrationRequest( - chatId="test-session", - message=request.message, - authorId="test-user", - conversationHistory=[], - url="test-context", - environment=request.environment, - connection_id=str(request.connectionId) if request.connectionId is not None else None, -) -``` - -- Calls `orchestration_service.process_orchestration_request(full_request)`. - -### 1.3 Orchestration Pipeline — Service Detection - -**`llm_orchestration_service.py` → `process_orchestration_request()`** (line 279): - -``` -STEP 0: Language detection → detect_language(request.message) → "en" - -STEP 0.1: Check request.message.startswith(SERVICE_STEP_PREFIXES) - → FALSE (natural language) → skip, continue normally - -STEP 0.5: Query validation → validate_query_basic(request.message) → valid - -STEP 1: Component initialization → LLM manager, guardrails adapter - -STEP 2: Input guardrails check → allowed - -STEP 3: ToolClassifier.classify(query, conversation_history, language) - → Classification(workflow=SERVICE, confidence=0.92) - The classifier uses hybrid search (dense + BM25) against - the intent_collections Qdrant collection. - -STEP 4: route_to_workflow(classification, request, is_streaming=False) - → Routes to ServiceWorkflowExecutor.execute_async() -``` - -### 1.4 ServiceWorkflowExecutor — `execute_async()` - -**`service_workflow.py` → `execute_async()`** (line 699): - -The executor uses **classification metadata** from hybrid search to decide how to proceed. There are three paths: - -| Condition | Path | -|---|---| -| `needs_llm_confirmation == False` | High-confidence match — run intent detection on the single top match only | -| `needs_llm_confirmation == True` | Ambiguous — run intent detection on top-N candidates | -| No metadata | Fall back to full discovery flow (`_log_request_details`) | - -#### Service Discovery (full flow) - -1. **`_call_service_discovery(chat_id)`** — calls `GET http://ruuter-public:8086/rag-search/services/get-services` -2. Checks if `service_count > SERVICE_COUNT_THRESHOLD (10)` → triggers **semantic search** via Qdrant -3. Otherwise uses the services list directly - -#### Intent Detection - -**`_process_intent_detection(services, request, chat_id, context, costs_metric)`**: -1. Calls **`_detect_service_intent()`** → uses `IntentDetectionModule` (DSPy LLM) with: - - The user query - - The candidate services list - - Conversation history -2. Returns matched `service_id`, `confidence`, `entities` -3. **`_validate_detected_service()`** — confirms the matched service exists in the active services list - -#### Entity Extraction & Validation - -```python -service_metadata = self._extract_service_metadata(context, chat_id) -# → {service_id, service_name, entities_dict, entity_schema, ruuter_type, is_common} - -validation_result = self._validate_entities(entities_dict, entity_schema, service_name, chat_id) -# → checks missing, extra, empty entities - -entities_array = self._transform_entities_to_array(entities_dict, entity_schema) -# → ordered list of entity values matching service schema -``` - -#### Service Endpoint Call - -```python -endpoint_url = self._construct_service_endpoint(service_name, chat_id, is_common) -# → "http://ruuter:8086/services/services/active/Klaviatuuri_probleemi_lahendamine" - -service_result = await self._call_service_endpoint( - endpoint_url, http_method, entities_array, chat_id, author_id -) -``` - -**`_call_service_endpoint()`** (line 523): -1. Sends POST/GET to the Ruuter endpoint with payload `{chatId, authorId, input: entities_array}` -2. Ruuter executes the DSL → DMapper produces the response -3. Parses the response: - - Unwraps `{"response": ...}` wrapper - - Extracts `data[0].content` → text content - - Extracts `data[0].buttons` → JSON string or list of `{title, payload}` objects -4. Returns `{"content": str, "buttons": List[Dict]}` - -#### Build Response - -```python -service_buttons = service_result["buttons"] -buttons_list = [ChoiceButton(**b) for b in service_buttons if "title" in b and "payload" in b] - -return OrchestrationResponse( - chatId=request.chatId, - llmServiceActive=True, - questionOutOfLLMScope=False, - inputGuardFailed=False, - content=service_content, - buttons=buttons_list if buttons_list else None, -) -``` - -### 1.5 API Layer — Response Conversion - -Back in `test_orchestrate_llm_request()` (line 382): - -```python -test_response = TestOrchestrationResponse( - llmServiceActive=response.llmServiceActive, - questionOutOfLLMScope=response.questionOutOfLLMScope, - inputGuardFailed=response.inputGuardFailed, - content=response.content, - buttons=response.buttons, # ← forwarded - chunks=None, -) -``` - -### 1.6 Frontend — Display Result - -**`TestModel/index.tsx`**: - -```tsx -// onSuccess callback (line 51-54) -setInferenceResult(data?.response); -setPendingButtons(data?.response?.buttons ?? []); -``` - -- Response text is rendered inside `` (line 166-168) -- MCQ buttons are rendered if `pendingButtons.length > 0` (line 173-186): - -```tsx -{pendingButtons.map((btn) => ( - -))} -``` - ---- - -## Flow 2: MCQ Button Click (Direct Step — Short-Circuit) - -### 2.1 Frontend — Button Click - -When the user clicks a button (e.g., *[Windows]*), `handleButtonClick()` is called (line 81): - -```tsx -const handleButtonClick = (payload: string) => { - if (!testLLM.connectionId) return; - setPendingButtons([]); - inferenceMutation.mutate({ - llmConnectionId: Number(testLLM.connectionId), - message: payload, // e.g. "#service, /POST/services/active/Klaviatuuri_probleemi_lahendamine_mcq_1_0" - }); -}; -``` - -The button **payload** becomes the next message. The same API call (`/rag-search/inference/test` → `/orchestrate/test`) is made. - -### 2.2 Input Sanitizer Safety - -The `#service, /POST/...` payload goes through Pydantic's `validate_and_sanitize_message()` on `OrchestrationRequest.message` (line 64-88 of `request_models.py`). The `InputSanitizer.sanitize_message()` strips HTML tags and normalizes whitespace but leaves `#`, `,`, `/` characters intact. The payload passes through unchanged. - -### 2.3 Orchestration — `#service` Prefix Short-Circuit - -**`process_orchestration_request()`** (line 324-340): - -```python -# STEP 0.1: Multi-step service prefix check (bypass NLU pipeline) -if request.message.startswith(SERVICE_STEP_PREFIXES): - logger.info(f"[{request.chatId}] #service prefix detected - direct step execution") - executor = self._get_service_workflow_executor() - direct_response = await executor.execute_direct_step( - request=request, - time_metric=time_metric, - ) - if direct_response is not None: - log_step_timings(time_metric, request.chatId) - return direct_response -``` - -**`SERVICE_STEP_PREFIXES`** = `("#service,", "#common_service,")` from `constants.py`. - -**`_get_service_workflow_executor()`** (line 264): -- Reuses the existing `tool_classifier.service_workflow` if a ToolClassifier has been initialized -- Otherwise creates a lightweight `ServiceWorkflowExecutor(llm_manager=None, orchestration_service=self)` — no LLM is needed for direct steps - -### 2.4 ServiceWorkflowExecutor — `execute_direct_step()` - -**`service_workflow.py` → `execute_direct_step()`** (line 1000): - -1. **Parse**: `_parse_service_prefix(request.message)` - - Input: `"#service, /POST/services/active/Klaviatuuri_probleemi_lahendamine_mcq_1_0"` - - Splits off prefix → remainder: `/POST/services/active/...` - - Extracts HTTP method: `POST` - - Builds URL: `http://ruuter:8086/services/services/active/Klaviatuuri_probleemi_lahendamine_mcq_1_0` - - Returns: `("POST", "http://ruuter:8086/services/services/active/...")` - -2. **Call endpoint**: `_call_service_endpoint(url, "POST", [], chat_id, author_id)` - - `entities_array=[]` — no entities for MCQ steps - - Same parsing as Flow 1 (extracts `content` + `buttons`) - -3. **Build response**: Same `OrchestrationResponse` construction as Flow 1 - -### What Gets Skipped (Short-Circuit) - -| Skipped Step | Why | -|---|---| -| Query validation | Would reject `#service` as gibberish | -| Component initialization | Expensive (LLM manager, Vault, guardrails) | -| Input guardrails | Would block a machine-generated payload | -| ToolClassifier.classify() | LLM call — unnecessary cost | -| Intent detection LLM | Another LLM call — URL is already known | -| Entity extraction | No natural language entities to extract | -| Semantic search (Qdrant) | No need to find a service | - -### 2.5 Response & Loop - -The response follows the same path back through `test_orchestrate_llm_request()` → frontend. - -- If the response has `buttons` → frontend renders the next set of MCQ buttons -- If the response has `buttons=null` → the MCQ flow is complete, only the final text answer is shown - ---- - -## Data Models - -### Request - -```python -class TestOrchestrationRequest: - message: str - environment: Literal["production", "testing", "development"] - connectionId: Optional[int] -``` - -### Response - -```python -class TestOrchestrationResponse: - llmServiceActive: bool - questionOutOfLLMScope: bool - inputGuardFailed: bool - content: str - buttons: Optional[List[ChoiceButton]] # MCQ buttons - chunks: Optional[List[ChunkInfo]] # RAG context chunks - -class ChoiceButton: - title: str # "Windows" - payload: str # "#service, /POST/services/active/..." -``` - ---- - -## Complete MCQ Sequence Diagram - -``` -User TestModel UI API (/orchestrate/test) ServiceWorkflow Ruuter/DMapper - │ │ │ │ │ - │ types "keyboard │ │ │ │ - │ not working" │ │ │ │ - ├───────────────────▶│ POST inference/test │ │ │ - │ ├────────────────────────▶│ process_orchestration_ │ │ - │ │ │ request() │ │ - │ │ │ startswith(#service)?→NO │ │ - │ │ │ classify()→SERVICE │ │ - │ │ ├───────────────────────────▶│ execute_async() │ - │ │ │ │ intent detect→matched │ - │ │ │ │ _call_service_endpoint │ - │ │ │ ├──────────────────────▶│ - │ │ │ │◁ {content,buttons}─────│ - │ │ │◁─ OrchestrationResponse ───│ │ - │ │◁─ TestOrchResponse ─────│ │ │ - │◁─ render text + │ │ │ │ - │ [Windows] [Mac] │ │ │ │ - │ │ │ │ │ - │ clicks [Windows] │ │ │ │ - ├───────────────────▶│ POST inference/test │ │ │ - │ │ msg="#service,/POST/…" │ │ │ - │ ├────────────────────────▶│ startswith(#service)?→YES │ │ - │ │ │ SKIP classifier+guardrails│ │ - │ │ ├───────────────────────────▶│ execute_direct_step() │ - │ │ │ │ _parse_service_prefix │ - │ │ │ │ _call_service_endpoint │ - │ │ │ ├──────────────────────▶│ - │ │ │ │◁ {content,buttons}─────│ - │ │ │◁─ OrchestrationResponse ───│ │ - │ │◁─ TestOrchResponse ─────│ │ │ - │◁─ render next MCQ │ │ │ │ - │ or final answer │ │ │ │ -``` - ---- - -## Error Handling - -| Error Scenario | Behavior | -|---|---| -| Service discovery fails | `execute_async()` returns `None` → falls back to RAG/context pipeline | -| Intent detection fails | Returns `None` → falls back to RAG/context pipeline | -| `_parse_service_prefix()` fails | `execute_direct_step()` returns `None` → falls through to normal pipeline | -| Service endpoint timeout | `_call_service_endpoint()` returns `None` → falls back | -| Service endpoint HTTP error | Logged, returns `None` → falls back | -| Buttons JSON parsing fails | Logs warning, returns empty buttons list `[]` | diff --git a/docs/TOOL_CLASSIFIER.md b/docs/TOOL_CLASSIFIER.md new file mode 100644 index 00000000..d3e04dbf --- /dev/null +++ b/docs/TOOL_CLASSIFIER.md @@ -0,0 +1,154 @@ +# Tool Classifier + +## Overview + +The **Tool Classifier** is the entry router of the LLM Module. For every incoming query it inspects the +message (and conversation state) and dispatches it to exactly one **workflow** that knows how to answer. +Retrieval-Augmented Generation (RAG) is just one of those workflows. + +This document gives a high-level understanding of the classifier itself — its routing model, key +functions, and configuration. The **behaviour of each individual workflow is documented separately**; +see [Related documentation](#related-documentation). + +Source: [`src/tool_classifier/`](../src/tool_classifier/). The classifier only runs when +`TOOL_CLASSIFIER_ENABLED=true`; otherwise the service uses the RAG-only pipeline (backward compatible). + +--- + +## Routing model + +Workflows are evaluated as a **layer-wise chain** (Strategy pattern). Each workflow either handles the +query or returns `None` to fall through to the next layer. The order is defined by +`WORKFLOW_LAYER_ORDER` in [`enums.py`](../src/tool_classifier/enums.py): + +``` +User query + │ + ├─ active API-tool session for this chat_id? ──► short-circuit to API_TOOL_CALLING + │ + ▼ +Layer 1: SERVICE → external Bürokratt service calls + ↓ (None) +Layer 2: API_TOOL_CALLING → agentic external API tool calling + ↓ (None) +Layer 3: CONTEXT → greetings + conversation-history answers + ↓ (None) +Layer 4: RAG → knowledge-base retrieval + generation + ↓ (None) +Layer 5: OOD → out-of-domain fallback (always answers) +``` + +If the classifier errors at any point, it falls back to RAG (`FALLBACK_TO_RAG_ON_ERROR = True`). + +--- + +## How classification works + +`classify()` uses a **two-step search** to decide the workflow: + +1. **Dense search** — cosine similarity against the service collection for a relevance check. +2. **Hybrid search** — dense + sparse (BM25) vectors fused with Reciprocal Rank Fusion (RRF) to + identify the best-matching service. + +High-confidence matches route straight to `SERVICE`; if no service matches, an API-tool search may route +to `API_TOOL_CALLING`; otherwise the query falls through to `CONTEXT` / `RAG`. The full scoring scheme +(thresholds, score-gap logic, sparse encoding) is documented in +[HYBRID_SEARCH_CLASSIFICATION.md](./HYBRID_SEARCH_CLASSIFICATION.md). + +--- + +## Key functions + +### `ToolClassifier` ([`classifier.py`](../src/tool_classifier/classifier.py)) + +| Method | Purpose | +| --- | --- | +| `classify(query, conversation_history, language, request=None)` | Runs the two-step search and returns a `ClassificationResult` indicating the target workflow. Also handles the active-session short-circuit and intent-switch detection. | +| `route_to_workflow(classification, request, is_streaming, ...)` | Executes the chosen workflow with layer-wise fallback. Returns an `OrchestrationResponse` (non-streaming) or an SSE `AsyncIterator[str]` (streaming). | +| `aclose()` | Releases the shared Qdrant `httpx` client. | + +Internal helpers (not part of the public surface): `_dense_search()`, `_hybrid_search()`, +`_try_api_tool_classification()`, and `_execute_with_fallback_async/streaming()`. + +### `BaseWorkflow` ([`base_workflow.py`](../src/tool_classifier/base_workflow.py)) + +Every workflow executor inherits this contract: + +| Method | Purpose | +| --- | --- | +| `execute_async(request, context, time_metric=None)` | Non-streaming execution (`/orchestrate`, `/orchestrate/test`). Returns a response, or `None` to fall back to the next layer. | +| `execute_streaming(request, context, time_metric=None)` | Streaming execution (`/orchestrate/stream`). Returns an SSE `AsyncIterator[str]`, or `None` to fall back. | + +The **return-`None` fallback** is the mechanism that powers the layer chain. + +### `ClassificationResult` ([`models.py`](../src/tool_classifier/models.py)) + +| Field | Type | Description | +| --- | --- | --- | +| `workflow` | `WorkflowType` | Which workflow should handle the query. | +| `confidence` | `float` (0.0–1.0) | Confidence in the classification. | +| `metadata` | `dict` | Workflow-specific data passed to the executor (e.g. matched service/endpoint). | +| `reasoning` | `str \| None` | Human-readable explanation of the decision. | + +--- + +## Workflow executors + +Each `WorkflowType` maps to one executor under +[`src/tool_classifier/workflows/`](../src/tool_classifier/workflows/). Detailed behaviour lives in the +linked docs. + +| Workflow | Executor | Detailed documentation | +| --- | --- | --- | +| `SERVICE` | `service_workflow.py` | [TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md](./TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md) | +| `API_TOOL_CALLING` | `api_tool_workflow.py` | [API_TOOL_CALLING.md](./API_TOOL_CALLING.md) | +| `CONTEXT` | `context_workflow.py` | [CONTEXT_WORKFLOW_GREETING_DETECTION.md](./CONTEXT_WORKFLOW_GREETING_DETECTION.md) | +| `RAG` | `rag_workflow.py` | [CONTEXTUAL_RETRIEVAL_FLOW.md](./CONTEXTUAL_RETRIEVAL_FLOW.md) | +| `OOD` | `ood_workflow.py` | — (fixed out-of-domain response) | + +--- + +## Configuration + +### Feature flags ([`feature_flags.py`](../src/llm_orchestrator_config/feature_flags.py)) + +All are environment variables read at startup. + +| Flag | Default | Effect | +| --- | --- | --- | +| `TOOL_CLASSIFIER_ENABLED` | `false` | Master switch. When `false`, the service uses the RAG-only pipeline. | +| `SERVICE_WORKFLOW_ENABLED` | `true` | Enables Layer 1 (Service). | +| `API_TOOL_CALLING_WORKFLOW_ENABLED` | `true` | Enables Layer 2 (API tool calling). | +| `CONTEXT_WORKFLOW_ENABLED` | `true` | Enables Layer 3 (Context). | +| `MULTI_INTENT_ENABLED` | `true` | Enables the parallel multi-intent path (IntentDecomposer) in API tool calling. | +| `ATC_RESPONSE_CACHE_ENABLED` | `true` | Enables the two-tier Redis response cache for API tool calling. | +| `FALLBACK_TO_RAG_ON_ERROR` | `true` (constant) | Routes to RAG if the classifier raises. | + +> RAG and OOD have no flags — RAG is the core fallback and OOD is the final safety net. + +### Classification constants ([`constants.py`](../src/tool_classifier/constants.py)) + +| Constant | Value | Purpose | +| --- | --- | --- | +| `QDRANT_COLLECTION` | `intent_collections` | Qdrant collection for Bürokratt services. | +| `API_TOOL_COLLECTION` | `api_tool_collection` | Qdrant collection for registered API tool endpoints. | +| `DENSE_MIN_THRESHOLD` | `0.5` | Below this cosine → not a service match. | +| `DENSE_HIGH_CONFIDENCE_THRESHOLD` | `0.55` | At/above (with gap) → SERVICE without LLM confirmation. | +| `DENSE_SCORE_GAP_THRESHOLD` | `0.05` | Required lead of the top service over the runner-up. | +| `API_TOOL_MIN_THRESHOLD` | `0.40` | Below this → no API-tool match. | +| `API_TOOL_HIGH_CONFIDENCE_THRESHOLD` | `0.60` | At/above → API-tool single-path immediately. | +| `API_TOOL_INTENT_SWITCH_THRESHOLD` | `0.50` | Min cosine for a new endpoint to interrupt an active session. | + +See [HYBRID_SEARCH_CLASSIFICATION.md](./HYBRID_SEARCH_CLASSIFICATION.md) for how these thresholds combine, +and [API_TOOL_CALLING.md](./API_TOOL_CALLING.md) for the multi-intent / caching constants. + +--- + +## Related documentation + +- [Hybrid Search Classification](./HYBRID_SEARCH_CLASSIFICATION.md) — scoring, sparse encoding, intent enrichment. +- [Service Workflow](./TOOL_CLASSIFIER_AND_SERVICE_WORKFLOW.md) — Layer 1 in depth. +- [API Tool Calling](./API_TOOL_CALLING.md) — Layer 2 agentic loop, multi-intent, response cache. +- [Context Workflow](./CONTEXT_WORKFLOW_GREETING_DETECTION.md) — Layer 3 greetings & history. +- [Contextual Retrieval Flow](./CONTEXTUAL_RETRIEVAL_FLOW.md) — Layer 4 RAG. +- [Architecture](./ARCHITECTURE.md) · [Documentation Index](./README.md) diff --git a/docs/TOOL_CLASSIFIER_EXTENSION_SPEC.md b/docs/TOOL_CLASSIFIER_EXTENSION_SPEC.md deleted file mode 100644 index 38d81898..00000000 --- a/docs/TOOL_CLASSIFIER_EXTENSION_SPEC.md +++ /dev/null @@ -1,1940 +0,0 @@ -# Tool Classifier Extension - System Specification - -**Version**: 1.0 -**Date**: February 13, 2026 -**Status**: Design Specification - ---- - -## 1. Overview - -This document specifies the extension of the existing RAG Module with a **Tool Classifier** that implements layer-wise workflow routing. The classifier determines whether a user query should be handled by: - -1. **Service Workflow** - External service/API calls -2. **Context Workflow** - Conversation history-based responses -3. **RAG Workflow** - Knowledge base retrieval (existing) -4. **OOD Response** - Out of domain fallback - -### 1.1 Current State - -**Existing Flow:** -``` -User Query → Input Guardrails → Prompt Refiner → Contextual Retrieval → Response Generator → Output Guardrails -``` - -**Entry Points:** -- `POST /orchestrate` - Non-streaming orchestration -- `POST /orchestrate/test` - Testing environment with simplified input -- `POST /orchestrate/stream` - Server-sent events streaming - -### 1.2 Proposed Extension - -**New Flow:** -``` -User Query → Input Guardrails → Tool Classifier → [Service | Context | RAG | OOD] - ↓ - Layer 1: Service Check - ↓ (no match) - Layer 2: Context Check - ↓ (no match) - Layer 3: RAG Retrieval - ↓ (no chunks) - Layer 4: OOD Response -``` - ---- - -## 2. Architecture Changes - -### 2.1 Component Integration - -The Tool Classifier will be integrated into the existing `LLMOrchestrationService` with minimal disruption: - -```python -# Location: src/llm_orchestration_service.py - -def process_orchestration_request(self, request: OrchestrationRequest): - """ - Modified orchestration pipeline with tool classifier. - - Pipeline: - 1. Language Detection (existing) - 2. Query Validation (existing) - 3. Input Guardrails (existing, relocated) - 4. Tool Classifier (NEW) - 5. Workflow Routing (NEW) - """ - - # Existing: Step 0, 0.5 - detected_language = detect_language(request.message) - validation_result = validate_query_basic(request.message) - - # Existing: Component initialization - components = self._initialize_service_components(request) - - # Existing: Step 1 - Input Guardrails (RELOCATED before classifier) - if components["guardrails_adapter"]: - input_blocked = self.handle_input_guardrails(...) - if input_blocked: - return input_blocked - - # NEW: Step 2 - Tool Classifier - classifier_result = self.tool_classifier.classify( - query=request.message, - conversation_history=request.conversationHistory, - language=detected_language - ) - - # NEW: Step 3 - Workflow Routing - if classifier_result.workflow == WorkflowType.SERVICE: - return self._execute_service_workflow(request, classifier_result) - elif classifier_result.workflow == WorkflowType.CONTEXT: - return self._execute_context_workflow(request, classifier_result) - elif classifier_result.workflow == WorkflowType.RAG: - return self._execute_rag_workflow(request, classifier_result) - else: - return self._create_out_of_scope_response(request, detected_language) -``` - -### 2.2 New Components - -| Component | Location | Purpose | -|-----------|----------|---------| -| `ToolClassifier` | `src/tool_classifier/classifier.py` | Main classifier logic | -| `ServiceWorkflowExecutor` | `src/tool_classifier/service_workflow.py` | Service discovery and triggering | -| `ContextWorkflowExecutor` | `src/tool_classifier/context_workflow.py` | LLM-based conversation history analysis | -| `IntentEntityExtractor` | `src/tool_classifier/intent_extractor.py` | LLM-based intent/entity detection | -| `ServiceDiscoveryManager` | `src/tool_classifier/service_discovery.py` | Qdrant semantic search for services | -| `IntentCollectionSync` | `src/tool_classifier/intent_sync_service.py` | Database → Qdrant synchronization | -| `ContextAnalyzer` | `src/tool_classifier/context_analyzer.py` | LLM-based context availability checker | - -### 2.3 LLM Config Module Integration - -The existing LLM Config Module (`src/llm_config_module/`) is reused by the tool classifier for all LLM-based operations. No modifications to the core module are required. - -**Current LLM Config Module Capabilities:** -- **Multi-Provider Support**: Azure OpenAI, AWS Bedrock, OpenAI, Anthropic -- **Vault Integration**: Secure credential management via HashiCorp Vault -- **Connection Management**: Dynamic LLM connection selection based on `connection_id` from requests -- **Usage Tracking**: Token counting and cost calculation across providers - -**Tool Classifier LLM Usage:** - -| Workflow | LLM Operation | Config Usage | Temperature | -|----------|---------------|--------------|-------------| -| **Service (Layer 1)** | Intent & entity extraction | `llm_manager.call_llm_async()` | 0.0 (deterministic) | -| **Context (Layer 2)** | Context availability check | `llm_manager.call_llm_async()` | 0.0 (deterministic) | -| **RAG (Layer 3)** | Response generation | Existing integration | 0.7 (default) | -| **OOD (Layer 4)** | No LLM call | N/A | N/A | - -**Integration Pattern:** - -```python -# Tool classifier workflows use the same LLMManager instance -class ToolClassifier: - def __init__(self, llm_manager: LLMManager, ...): - self.llm_manager = llm_manager # Reuse existing instance - - async def detect_intent(self, query: str, services: List[Service]): - """Use LLM Config Module for intent detection.""" - response = await self.llm_manager.call_llm_async( - prompt=INTENT_DETECTION_PROMPT.format(...), - temperature=0.0, # Deterministic for classification - max_tokens=200 - ) - return parse_intent(response) -``` - -**Configuration Reuse:** -- Same connection selection logic (`connection_id` from `OrchestrationRequest`) -- Same Vault credential retrieval -- Same cost tracking pattern (`get_lm_usage_since()`) -- Same error handling and retry logic -- Same provider-specific implementations - -**No Changes Required**: The LLM Config Module is provider-agnostic and supports all tool classifier LLM calls out of the box. - ---- - -## 3. Layer 1: Service Workflow - -### 3.1 Workflow Logic - -When a user query is received, the system determines if it's a service-related request through the following steps: - -``` -1. Service Count Check → 2. Service Discovery → 3. Intent Detection → 4. Service Validation → 5. Entity Transformation → 6. Service Triggering -``` - -### 3.2 Step-by-Step Implementation - -#### Step 1: Service Count Check - -**Purpose**: Optimize performance based on service catalog size - -```python -# Query: SELECT COUNT(*) FROM services WHERE current_state = 'active' AND deleted = FALSE - -if service_count <= 50: - # Use all services for LLM context - services = get_all_active_services() -else: - # Use semantic search for top 20 most relevant - services = semantic_search_services(user_query, top_k=20) -``` - -**Database Query:** -```sql -SELECT COUNT(*) FROM public.services -WHERE current_state = 'active' AND deleted = FALSE; -``` - -#### Step 2: Semantic Search (When Service Count > 50) - -**Tool**: Qdrant vector database -**Collection**: `intent_collection` -**Vector Dimension**: 3072 (text-embedding-3-large) - -**Search Configuration:** -```python -search_params = { - "collection_name": "intent_collection", - "query_vector": embed_query(user_query), - "limit": 20, - "score_threshold": 0.5, # Higher threshold for service matching -} -``` - -**Output Format:** -```json -[ - { - "service_id": "exchange-rate-001", - "service_name": "ExchangeRateService", - "description": "Provides currency exchange rates", - "entities": ["fromCurrency", "toCurrency"], - "score": 0.87 - }, - ... -] -``` - -#### Step 3: LLM Intent Detection - -**Action**: Call LLM with user query and service context to extract: -- `intent`: Service name to trigger -- `entities`: Key-value pairs of extracted parameters - -**Prompt Template:** -```python -INTENT_DETECTION_PROMPT = """ -You are an intent classifier for government services. Analyze the user query and determine which service should handle the request. - -Available Services: -{service_list} - -User Query: "{user_query}" - -Task: -1. If the query matches a service, extract: - - intent: The exact service name to trigger - - entities: Key-value pairs of required parameters - -2. If NO service matches, respond with: {{"intent": null, "entities": null}} - -Response Format (JSON only, no explanation): -{{"intent": "ServiceName", "entities": {{"param1": "value1", "param2": "value2"}}}} -""" -``` - -**Expected LLM Response:** -```json -{ - "choices": [ - { - "message": { - "content": "{\"intent\": \"ExchangeRateService\", \"entities\": {\"fromCurrency\": \"EUR\", \"toCurrency\": \"USD\"}}" - } - } - ] -} -``` - -**Parsing Logic:** -```python -# Parse LLM response -content = response["choices"][0]["message"]["content"] -parsed = json.loads(content) - -if parsed["intent"] is None: - # No service match - move to Layer 2 (Context Workflow) - return WorkflowType.CONTEXT -``` - -#### Step 4: Service Validation - -**Action**: Validate the detected service against the database - -**Validation Query:** -```sql -SELECT service_id, name, ruuter_type, endpoints, structure, entities -FROM public.services -WHERE service_id = %(detected_service_id)s - AND current_state = 'active' - AND deleted = FALSE; -``` - -**Validation Checks:** -- Service exists in database -- `current_state = 'active'` -- `deleted = FALSE` - -**Failure Handling:** -```python -if not service_exists or not service_active: - logger.warning(f"Service validation failed: {detected_service_id}") - # Fallback to Layer 2 (Context Workflow) - return WorkflowType.CONTEXT -``` - -#### Step 5: Entity Transformation - -**Purpose**: Convert LLM entity object to array format for service payload - -**Input (from LLM):** -```json -{ - "fromCurrency": "EUR", - "toCurrency": "USD" -} -``` - -**Output (for service call):** -```json -["EUR", "USD"] -``` - -**Transformation Logic:** -```python -def transform_entities(entities: Optional[Dict[str, str]], - entity_order: List[str]) -> List[str]: - """ - Transform entity dictionary to ordered array. - - Args: - entities: LLM-extracted entity key-value pairs - entity_order: Expected entity order from service schema - - Returns: - Ordered list of entity values - """ - if not entities or entities is None: - return [] - - # Maintain order defined in service schema - return [entities.get(key, "") for key in entity_order] -``` - -**Example:** -```python -# Service schema defines: entities = ["fromCurrency", "toCurrency"] -transform_entities( - {"fromCurrency": "EUR", "toCurrency": "USD"}, - ["fromCurrency", "toCurrency"] -) -# Output: ["EUR", "USD"] -``` - -#### Step 6: Service Triggering - -**Purpose**: Call the external service endpoint with formatted payload - -**URL Construction:** -```python -# From database field 'endpoints' -base_url = "http://ruuter:8086" # From environment or service config -service_endpoint = f"{base_url}/services/active{service_name}" - -# Example: http://ruuter:8086/services/activeExchangeRateService -``` - -**HTTP Method:** -```python -# Retrieved from database field 'ruuter_type' -method = service.ruuter_type # 'GET' or 'POST' (ENUM) -``` - -**Payload Format:** -```json -{ - "input": ["EUR", "USD"], - "authorId": "user-67890", - "chatId": "chat-12345" -} -``` - -**Implementation:** -```python -async def trigger_service( - service: ServiceRecord, - entities: List[str], - request: OrchestrationRequest -) -> Dict[str, Any]: - """ - Trigger external service via Ruuter. - - Args: - service: Validated service record from database - entities: Transformed entity array - request: Original orchestration request - - Returns: - Service response or error - """ - url = f"{RUUTER_BASE_URL}/services/active{service.name}" - payload = { - "input": entities, - "authorId": request.authorId, - "chatId": request.chatId - } - - try: - if service.ruuter_type == "GET": - response = await http_client.get(url, params=payload, timeout=10) - else: # POST - response = await http_client.post(url, json=payload, timeout=10) - - response.raise_for_status() - return response.json() - - except httpx.TimeoutException: - logger.error(f"Service timeout: {service.service_id}") - raise ServiceTimeoutError() - except httpx.HTTPStatusError as e: - logger.error(f"Service error: {e.response.status_code}") - raise ServiceExecutionError() -``` - -**Response Handling:** - -**Non-Streaming:** -```python -service_response = await trigger_service(service, entities, request) -formatted_content = format_service_response(service_response) - -# Apply output guardrails -if guardrails_adapter: - output_check = await guardrails_adapter.check_output_async(formatted_content) - costs_metric["output_guardrails"] = output_check.usage - - if not output_check.allowed: - logger.warning(f"Service response blocked by guardrails: {output_check.reason}") - return create_guardrail_violation_response(request) - -# Return validated service response -return OrchestrationResponse( - chatId=request.chatId, - llmServiceActive=True, - questionOutOfLLMScope=False, - inputGuardFailed=False, - content=formatted_content -) -``` - -**Streaming:** -```python -service_response = await trigger_service(service, entities, request) -formatted_content = format_service_response(service_response) - -# Apply output guardrails validation -if guardrails_adapter: - output_check = await guardrails_adapter.check_output_async(formatted_content) - costs_metric["output_guardrails"] = output_check.usage - - if not output_check.allowed: - logger.warning(f"Service response blocked by guardrails") - yield format_sse(request.chatId, OUTPUT_GUARDRAIL_VIOLATION_MESSAGE) - yield format_sse(request.chatId, "END") - return - -# Stream validated response token-by-token -for token in split_into_tokens(formatted_content, chunk_size=5): - yield format_sse(request.chatId, token) - await asyncio.sleep(0.01) # Maintain streaming UX - -yield format_sse(request.chatId, "END") -``` - -### 3.3 Failure Scenarios - -| Scenario | Action | -|----------|--------| -| No intent detected | Move to Layer 2 (Context Workflow) | -| Service validation failed | Move to Layer 2 (Context Workflow) | -| Service call timeout | Return `SERVICE_TIMEOUT_ERROR` message | -| Service returns error | Return `SERVICE_EXECUTION_ERROR` message | -| Entity extraction incomplete | Attempt service call with partial entities, or fallback to Layer 2 | -| Output guardrails blocked | Return `OUTPUT_GUARDRAIL_VIOLATION_MESSAGE` or fallback to Layer 2 | - -### 3.4 Output Guardrails for Service Responses - -**Why Service Responses Need Guardrails:** -- External services may return PII (personal identifiable information) -- Service errors could expose sensitive system details -- Third-party API responses are untrusted content -- Ensures consistent safety across all workflows - -**Integration Pattern:** - -Both non-streaming and streaming modes validate service responses before sending to users: - -```python -# Get service response -service_response = await trigger_service(...) - -# Apply output guardrails (validation-first) -if guardrails_adapter: - output_check = await guardrails_adapter.check_output_async(service_response) - if not output_check.allowed: - # Blocked - return error or fallback - return create_guardrail_violation_response(request) - -# Validated - return/stream to user -return/stream service_response -``` - ---- - -## 4. Layer 2: Context Workflow - -### 4.1 Workflow Logic - -If Layer 1 fails (no service match), use LLM to determine if the query is a greeting or can be answered from conversation history. - -**Trigger Conditions:** -- No service intent detected in Layer 1 -- Query is a greeting (hello, hi, good morning, etc.) **OR** -- Conversation history exists (at least 1 previous turn) and query references it - -### 4.2 Greeting Detection - -Greetings and conversational pleasantries are handled by the Context Workflow to provide natural, friendly responses without triggering service discovery or RAG retrieval. - -**Greeting Patterns (Multilingual):** - -```python -# Estonian greetings -ESTONIAN_GREETINGS = [ - "tere", "tervist", "tere hommikust", "tere päevast", "tere õhtust", - "hei", "hommikust", "õhtust", "päevast", "nägemist", - "tsau", "moi", "moikka" -] - -# English greetings -ENGLISH_GREETINGS = [ - "hello", "hi", "hey", "good morning", "good afternoon", "good evening", - "greetings", "howdy", "morning", "afternoon", "evening" -] - -# Farewell patterns -FAREWELL_PATTERNS = [ - "goodbye", "bye", "see you", "talk to you later", "ttyl", - "nägemist", "head aega", "kuni", "tsau" -] -``` - -**LLM-Based Greeting Detection:** - -Instead of rigid pattern matching, the LLM analyzes whether the query is a greeting or conversational message: - -```python -async def detect_greeting( - query: str, - llm_manager: LLMManager, - language: str -) -> GreetingResult: - """ - Use LLM to detect if query is a greeting/conversational message. - - Args: - query: User's message - llm_manager: LLM manager instance - language: Detected language (et/en) - - Returns: - GreetingResult with is_greeting flag and optional response - """ - prompt = GREETING_DETECTION_PROMPT.format( - user_query=query, - language=language - ) - - response = await llm_manager.call_llm_async( - prompt=prompt, - temperature=0.0, - max_tokens=150 - ) - - content = response["choices"][0]["message"]["content"] - result = json.loads(content) - - return GreetingResult( - is_greeting=result["is_greeting"], - greeting_type=result.get("greeting_type"), # 'hello', 'goodbye', 'thanks', etc. - suggested_response=result.get("suggested_response") - ) -``` - -**Greeting Detection Prompt:** - -```python -GREETING_DETECTION_PROMPT = """ -You are a greeting classifier. Determine if the user's message is a greeting, farewell, or conversational pleasantry. - -User Message: "{user_query}" -Language: {language} - -Task: -1. Identify if this is a greeting/conversational message (hello, hi, goodbye, thanks, etc.) -2. If YES: Classify the type and suggest an appropriate response -3. If NO: Indicate it's not a greeting - -Response Format (JSON only): -{{ - "is_greeting": true/false, - "greeting_type": "hello" | "goodbye" | "thanks" | "casual" | null, - "suggested_response": "friendly response in same language" | null -}} - -Examples of greetings: -- "Tere!" → {"is_greeting": true, "greeting_type": "hello"} -- "Good morning" → {"is_greeting": true, "greeting_type": "hello"} -- "Thanks for your help" → {"is_greeting": true, "greeting_type": "thanks"} -- "What are digital signatures?" → {"is_greeting": false} -""" -``` - -**Response Generation:** - -```python -if greeting_result.is_greeting: - # Use LLM-suggested response or fallback to predefined messages - response = greeting_result.suggested_response or get_default_greeting_response( - greeting_type=greeting_result.greeting_type, - language=language - ) - - return OrchestrationResponse( - chatId=request.chatId, - llmServiceActive=True, - questionOutOfLLMScope=False, - inputGuardFailed=False, - content=response - ) -``` - -### 4.3 LLM-Based Context Analysis - -Instead of using regex patterns, we use the LLM to intelligently determine if the query references conversation history and can be answered from it. - -**Conversation Window:** -```python -# Consider last 10 conversation turns (5 user + 5 bot pairs) -CONTEXT_WINDOW_SIZE = 10 - -def get_recent_history(history: List[ConversationItem]) -> List[ConversationItem]: - """Get recent conversation history for context analysis.""" - return history[-CONTEXT_WINDOW_SIZE:] if history else [] -``` - -**LLM Context Check Prompt:** -```python -CONTEXT_CHECK_PROMPT = """ -You are a conversation context analyzer. Analyze if the user's current query can be answered using ONLY the conversation history provided. - -Conversation History: -{conversation_history} - -Current User Query: "{user_query}" - -Task: -1. First check if this is a greeting/conversational message (hi, hello, thanks, goodbye, etc.) -2. If it's a greeting: Provide an appropriate friendly response -3. If NOT a greeting: Determine if the query references or can be answered from the conversation history above -4. If YES: Extract and provide the answer from the conversation history -5. If NO: Indicate that it cannot be answered from conversation history - -Response Format (JSON only, no explanation): -{{ - "is_greeting": true/false, - "can_answer_from_context": true/false, - "answer": "extracted answer from history OR greeting response" OR null, - "reasoning": "brief explanation of why it can/cannot be answered" -}} - -Examples of GREETINGS (handle with friendly response): -- "Tere!" → {"is_greeting": true, "answer": "Tere! Kuidas saan teid aidata?"} -- "Hello" → {"is_greeting": true, "answer": "Hello! How can I help you?"} -- "Thanks!" → {"is_greeting": true, "answer": "You're welcome!"} -- "Good morning" → {"is_greeting": true, "answer": "Good morning! What can I do for you?"} - -Examples of queries that CAN be answered from context: -- "What did you say earlier about that?" -- "Can you repeat that?" -- "What was the rate you mentioned?" -- "Tell me more about what you just said" - -Examples of queries that CANNOT be answered from context: -- Completely new topics -- Requests for real-time data -- Questions requiring external knowledge -""" -``` - -**Implementation:** -```python -async def check_context_availability( - query: str, - conversation_history: List[ConversationItem], - llm_manager: LLMManager -) -> ContextCheckResult: - """ - Use LLM to check if query can be answered from conversation history. - - Args: - query: Current user query - conversation_history: Recent conversation turns - llm_manager: LLM manager for making calls - - Returns: - ContextCheckResult with can_answer flag and optional answer - """ - # Get recent history - recent_history = get_recent_history(conversation_history) - - if not recent_history: - # No conversation history available - return ContextCheckResult( - can_answer_from_context=False, - answer=None, - reasoning="No conversation history available" - ) - - # Format conversation history for prompt - history_text = format_conversation_history(recent_history) - - # Call LLM with structured output request - prompt = CONTEXT_CHECK_PROMPT.format( - conversation_history=history_text, - user_query=query - ) - - try: - response = await llm_manager.call_llm_async( - prompt=prompt, - temperature=0.0, # Deterministic for classification - max_tokens=300 - ) - - # Parse structured JSON response - content = response["choices"][0]["message"]["content"] - result = json.loads(content) - - return ContextCheckResult( - is_greeting=result.get("is_greeting", False), - can_answer_from_context=result["can_answer_from_context"], - answer=result.get("answer"), - reasoning=result.get("reasoning", "") - ) - - except (json.JSONDecodeError, KeyError) as e: - logger.error(f"Failed to parse LLM context check response: {e}") - # Fallback: assume cannot answer from context - return ContextCheckResult( - can_answer_from_context=False, - answer=None, - reasoning="Failed to parse LLM response" - ) - -def format_conversation_history(history: List[ConversationItem]) -> str: - """Format conversation history for LLM prompt.""" - formatted = [] - for i, item in enumerate(history, 1): - role = "User" if item.authorRole == "user" else "Assistant" - formatted.append(f"{i}. {role}: {item.message}") - return "\n".join(formatted) -``` - -**Response Models:** -```python -from pydantic import BaseModel - -class ContextCheckResult(BaseModel): - """Result from LLM context availability check.""" - is_greeting: bool = False - can_answer_from_context: bool - answer: Optional[str] = None - reasoning: str = "" - -class GreetingResult(BaseModel): - """Result from greeting detection.""" - is_greeting: bool - greeting_type: Optional[str] = None # 'hello', 'goodbye', 'thanks', 'casual' - suggested_response: Optional[str] = None -``` - -### 4.3 Workflow Execution - -**Non-Streaming Response:** -```python -async def execute_context_workflow( - request: OrchestrationRequest, - llm_manager: LLMManager, - guardrails_adapter: Optional[NeMoRailsAdapter], - costs_metric: Dict -) -> Optional[OrchestrationResponse]: - """ - Execute context-based response workflow with output guardrails. - - Returns: - OrchestrationResponse with context-based answer or None to fallback to next layer - """ - # Check if query can be answered from conversation history - context_result = await check_context_availability( - query=request.message, - conversation_history=request.conversationHistory, - llm_manager=llm_manager - ) - - # Track costs - costs_metric["context_check"] = get_lm_usage_since(history_before) - - if (context_result.is_greeting or context_result.can_answer_from_context) and context_result.answer: - logger.info( - f"[{request.chatId}] Query answered from context " - f"(greeting: {context_result.is_greeting})" - ) - - # Apply output guardrails validation - if guardrails_adapter: - output_check = await guardrails_adapter.check_output_async( - context_result.answer - ) - costs_metric["output_guardrails"] = output_check.usage - - if not output_check.allowed: - logger.warning( - f"[{request.chatId}] Context response blocked by guardrails: " - f"{output_check.reason}" - ) - return create_guardrail_violation_response(request) - - # Return validated context-based response - return OrchestrationResponse( - chatId=request.chatId, - llmServiceActive=True, - questionOutOfLLMScope=False, - inputGuardFailed=False, - content=context_result.answer - ) - - else: - logger.info( - f"[{request.chatId}] Cannot answer from context: {context_result.reasoning}" - ) - # Fallback to Layer 3 (RAG Workflow) - return None # Signal to move to next layer -``` - -**Streaming Response:** -```python -async def execute_context_workflow_streaming( - request: OrchestrationRequest, - llm_manager: LLMManager, - guardrails_adapter: Optional[NeMoRailsAdapter], - costs_metric: Dict -) -> Optional[AsyncIterator[str]]: - """ - Execute context workflow with streaming support and output guardrails. - - Yields: - SSE-formatted strings with validated context-based response - - Returns: - None if cannot answer from context (signals fallback to next layer) - """ - # Check context availability (non-streaming, fast) - context_result = await check_context_availability( - query=request.message, - conversation_history=request.conversationHistory, - llm_manager=llm_manager - ) - - # Track costs - costs_metric["context_check"] = get_lm_usage_since(history_before) - - if (context_result.is_greeting or context_result.can_answer_from_context) and context_result.answer: - logger.info( - f"[{request.chatId}] Validating and streaming context-based response " - f"(greeting: {context_result.is_greeting})" - ) - - # Apply output guardrails validation BEFORE streaming - if guardrails_adapter: - output_check = await guardrails_adapter.check_output_async( - context_result.answer - ) - costs_metric["output_guardrails"] = output_check.usage - - if not output_check.allowed: - logger.warning( - f"[{request.chatId}] Context response blocked by guardrails (streaming)" - ) - yield format_sse(request.chatId, OUTPUT_GUARDRAIL_VIOLATION_MESSAGE) - yield format_sse(request.chatId, "END") - return - - # Response validated - stream token by token for consistent UX - for token in split_into_tokens(context_result.answer, chunk_size=5): - yield format_sse(request.chatId, token) - await asyncio.sleep(0.01) # Maintain streaming pace - - # Signal completion - yield format_sse(request.chatId, "END") - - else: - logger.info(f"[{request.chatId}] No context match, falling back to RAG") - # Return None to signal fallback to next layer - # Caller will handle RAG workflow - return None - -def split_into_tokens(text: str, chunk_size: int = 5) -> List[str]: - """Split text into token-like chunks for streaming simulation.""" - words = text.split() - tokens = [] - for i in range(0, len(words), chunk_size): - chunk = " ".join(words[i:i + chunk_size]) - tokens.append(chunk + " " if i + chunk_size < len(words) else chunk) - return tokens -``` - -### 4.4 Advantages of LLM-Based Approach - - **No Regex Pattern Maintenance**: LLM understands semantic context references naturally - **Handles Edge Cases**: Can detect implicit references that regex would miss - **Multilingual Support**: Works across Estonian, English, and other languages - **Structured Output**: Consistent JSON format for easy parsing - **Reasoning Transparency**: Includes explanation of decision - **Streaming Compatible**: Fast context check + token-by-token answer delivery - **Greeting Detection**: Automatically handles greetings, farewells, and conversational pleasantries - **Natural Responses**: LLM generates contextually appropriate greeting responses - -### 4.7 Fallback Strategy - -**Fallback to Layer 3 (RAG):** -- If `is_greeting = false` AND `can_answer_from_context = false` -- If LLM response parsing fails -- If conversation history is empty (and not a greeting) -- If output guardrails block the response (fallback to RAG for alternative answer) - -**Error Handling:** -```python -try: - result = await execute_context_workflow( - request, llm_manager, guardrails_adapter, costs_metric - ) - if result: - return result # Context-based answer (validated) - else: - # Move to Layer 3 (RAG) - return await execute_rag_workflow(request, components, costs_metric) -except Exception as e: - logger.error(f"Context workflow failed: {e}") - # Fallback to RAG workflow - return await execute_rag_workflow(request, components, costs_metric) -``` - -**Guardrail Violation Fallback:** -```python -# Option 1: Return error message (current approach) -if not output_check.allowed: - return create_guardrail_violation_response(request) - -# Option 2: Fallback to RAG (alternative approach) -if not output_check.allowed: - logger.warning("Context response blocked, trying RAG workflow") - return await execute_rag_workflow(request, components, costs_metric) -``` - ---- - -## 5. Layer 3: RAG Workflow - -### 5.1 Integration with Existing System - -**Trigger**: When both Layer 1 (Service) and Layer 2 (Context) fail to match - -**Implementation:** -```python -# Reuse existing RAG pipeline -return self._execute_orchestration_pipeline( - request, components, costs_metric, time_metric -) -``` - -**Existing Flow (No Changes Required):** -1. Prompt Refinement -2. Contextual Retrieval (Qdrant + BM25) -3. Rank Fusion (RRF) -4. Response Generation -5. Output Guardrails (validation-first streaming already implemented) - -**Streaming with Output Guardrails (Current Implementation):** -```python -# RAG workflow uses validation-first approach -async for validated_chunk in guardrails_adapter.stream_with_guardrails( - user_message=refined_query, - bot_message_generator=llm_streaming_generator -): - # NeMo buffers tokens (chunk_size=200) - # Validates each buffer before yielding - yield format_sse(chatId, validated_chunk) - -yield format_sse(chatId, "END") -``` - -**Fallback:** -- If no chunks found (`len(relevant_chunks) == 0`) → Layer 4 (OOD) -- If response confidence low → Layer 4 (OOD) - ---- - -## 5.2 Streaming + Output Guardrails Comparison - -### Summary: How Each Workflow Handles Streaming + Validation - -| Workflow | Response Source | Validation Approach | Streaming Method | -|----------|----------------|---------------------|------------------| -| **RAG** | LLM streaming generation | NeMo buffers + validates chunks (chunk_size=200) | `stream_with_guardrails()` wraps bot generator | -| **Service** | External service (complete) | Validate complete response | Stream validated response token-by-token | -| **Context** | LLM structured output (complete) | Validate complete response | Stream validated response token-by-token | -| **OOD** | Fixed message | No validation needed | Stream fixed message token-by-token | - -### Technical Flow for Each Workflow - -#### RAG Workflow (Existing - Validation-First) - -**Non-Streaming:** -```python -response = await response_generator.generate(...) -output_check = await guardrails_adapter.check_output_async(response) -if output_check.allowed: - return OrchestrationResponse(content=response) -``` - -**Streaming:** -```python -# LLM generates via streaming -async def bot_generator(): - async for token in llm.stream(): - yield token - -# NeMo validates in real-time (buffers chunks) -async for validated_chunk in guardrails_adapter.stream_with_guardrails( - user_message=query, - bot_message_generator=bot_generator -): - yield format_sse(chatId, validated_chunk) # Already validated -``` - -#### Service Workflow (New - Validate Then Stream) - -**Non-Streaming:** -```python -service_response = await call_external_service(...) # Complete response -output_check = await guardrails_adapter.check_output_async(service_response) -if output_check.allowed: - return OrchestrationResponse(content=service_response) -else: - return GuardrailViolationResponse() -``` - -**Streaming:** -```python -service_response = await call_external_service(...) # Complete response - -# Validate complete response FIRST -output_check = await guardrails_adapter.check_output_async(service_response) -if not output_check.allowed: - yield format_sse(chatId, OUTPUT_GUARDRAIL_VIOLATION_MESSAGE) - yield format_sse(chatId, "END") - return - -# Validated - now stream to client token-by-token -for token in split_into_tokens(service_response, chunk_size=5): - yield format_sse(chatId, token) - await asyncio.sleep(0.01) -yield format_sse(chatId, "END") -``` - -#### Context Workflow (New - Validate Then Stream) - -**Non-Streaming:** -```python -context_result = await llm.check_context(query, history) # Complete answer -if context_result.can_answer_from_context: - output_check = await guardrails_adapter.check_output_async(context_result.answer) - if output_check.allowed: - return OrchestrationResponse(content=context_result.answer) - else: - return GuardrailViolationResponse() -``` - -**Streaming:** -```python -context_result = await llm.check_context(query, history) # Complete answer - -if context_result.can_answer_from_context: - # Validate complete answer FIRST - output_check = await guardrails_adapter.check_output_async(context_result.answer) - if not output_check.allowed: - yield format_sse(chatId, OUTPUT_GUARDRAIL_VIOLATION_MESSAGE) - yield format_sse(chatId, "END") - return - - # Validated - stream to client token-by-token - for token in split_into_tokens(context_result.answer, chunk_size=5): - yield format_sse(chatId, token) - await asyncio.sleep(0.01) - yield format_sse(chatId, "END") -``` - -### Key Differences - -**RAG Workflow:** -- **Real-time validation**: LLM generates → NeMo validates chunks → Stream to client -- **Buffered approach**: Tokens buffered in chunks of 200 characters -- **Bi-directional**: Generator feeding into NeMo, NeMo yielding validated chunks -- **Cost**: Inline (no separate validation call) - -**Service/Context Workflows:** -- **Pre-validation**: Get complete response → Validate → Stream to client -- **Complete response**: Already have full text before streaming starts -- **Uni-directional**: Simply chunk and send validated response -- **Cost**: Separate validation call tracked in `costs_metric["output_guardrails"]` -- **UX Consistency**: Simulates streaming to match RAG workflow behavior - -### Why Different Approaches? - -1. **RAG**: LLM streaming is inherently token-by-token, so NeMo can validate in real-time -2. **Service**: External API returns complete response, no streaming generation occurs -3. **Context**: LLM returns structured JSON with complete answer, not streaming - -### Common Pattern: Validation-First - -All three workflows share the **validation-first principle**: -- Content is validated BEFORE reaching the user -- Blocked content never sent to client -- Consistent safety guarantees across all workflows -- Streaming provides smooth UX even with complete responses (Service/Context) - ---- - -## 6. Layer 4: OOD (Out of Domain) Response - -### 6.1 Trigger Conditions - -- No service detected (Layer 1 failed) -- No context match (Layer 2 failed) -- No relevant knowledge chunks (Layer 3 failed) - -### 6.2 Response Generation - -**Return localized OOD message:** -```python -return OrchestrationResponse( - chatId=request.chatId, - llmServiceActive=True, - questionOutOfLLMScope=True, # Flag as out of scope - inputGuardFailed=False, - content=get_localized_message(OUT_OF_SCOPE_MESSAGES, detected_language) -) -``` - -**Existing Constants (Reuse):** -```python -# From: src/llm_orchestrator_config/llm_ochestrator_constants.py -OUT_OF_SCOPE_MESSAGES = { - "et": "Vabandust, ma ei suuda sellele küsimusele vastata...", - "en": "I apologize, but I cannot answer this question..." -} -``` - ---- - -## 7. Data Schemas - -### 7.1 Database Schema - -**Table: `services`** - -```sql --- Location: DSL/Liquibase/changelog/rag-search-script-v6-services.sql - --- Custom ENUM types -CREATE TYPE ruuter_request_type AS ENUM ('GET', 'POST'); -CREATE TYPE service_state AS ENUM ('active', 'inactive', 'draft'); - -CREATE TABLE public.services ( - -- Primary key - id BIGINT PRIMARY KEY, - - -- Basic service information - name TEXT NOT NULL, -- Service name (e.g., "ExchangeRateService") - description TEXT NOT NULL, -- Human-readable description - service_id TEXT NOT NULL UNIQUE, -- Unique identifier (e.g., "exchange-rate-001") - - -- Service classification - ruuter_type ruuter_request_type DEFAULT 'GET', -- HTTP method: 'GET' or 'POST' - current_state service_state DEFAULT 'draft', -- State: 'active', 'inactive', 'draft' - is_common BOOLEAN NOT NULL DEFAULT FALSE, -- Is this a common/shared service? - deleted BOOLEAN NOT NULL DEFAULT FALSE, -- Soft delete flag - - -- Intent classification data (for LLM) - slot TEXT NOT NULL DEFAULT '', -- Reserved for future use - entities text[] NOT NULL DEFAULT '{}', -- Expected entity names ["entity1", "entity2"] - examples text[] NOT NULL DEFAULT '{}', -- Example queries - - -- Service configuration - structure JSON NOT NULL DEFAULT '{}', -- Service schema/structure - endpoints JSON NOT NULL DEFAULT '[]', -- Endpoint configurations - - -- Timestamps - created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, - updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP -); - --- Indexes for performance -CREATE UNIQUE INDEX idx_services_service_id ON public.services(service_id); -CREATE INDEX idx_services_active ON public.services(current_state, deleted) - WHERE deleted = FALSE; -CREATE INDEX idx_services_name ON public.services(name); -``` - -**Update Master Changelog:** -```yaml -# Location: DSL/Liquibase/master.yml - -databaseChangeLog: - - include: - file: changelog/rag-search-script-v1-llm-connections.sql - - include: - file: changelog/rag-search-script-v2-user-management.sql - - include: - file: changelog/rag-search-script-v3-configuration.sql - - include: - file: changelog/rag-search-script-v4-authority-data.xml - - include: - file: changelog/rag-search-script-v5-prompt-config.sql - - include: - file: changelog/rag-search-script-v6-services.sql # NEW -``` - -### 7.2 Qdrant Collection Schema - -**Collection Name:** `intent_collection` - -**Configuration:** -```python -{ - "collection_name": "intent_collection", - "vectors_config": { - "size": 3072, # text-embedding-3-large - "distance": "Cosine" - } -} -``` - -**Document Schema:** -```json -{ - "id": "common_service_companies_workforce_taxes", - "name": "Ettevõtte tööjõumaksud", - "description": "Kasutaja soovib infot ettevõtte poolt tasutud tööjõumaksude kohta, näiteks palgamaksud ja sotsiaalmaks.", - "examples": [ - "ettevõtte tasutud tööjõumaksud", - "kui palju maksis ettevõte tööjõumakse", - "firma poolt tasutud tööjõumaksud" - ], - "entities": ["company_name"], - "text_for_embedding": "Kasutaja soovib infot ettevõtte poolt tasutud tööjõumaksude kohta, näiteks palgamaksud ja sotsiaalmaks.\nettevõtte tasutud tööjõumaksud\nkui palju maksis ettevõte tööjõumakse\nfirma poolt tasutud tööjõumaksud", - - "service_id": "common_service_companies_workforce_taxes", - "ruuter_type": "POST", - "current_state": "active" -} -``` - -**Field Mapping:** -| Qdrant Field | Source | Purpose | -|--------------|--------|---------| -| `id` | `services.service_id` | Unique identifier | -| `name` | `services.name` | Service display name | -| `description` | `services.description` | Service description | -| `examples` | `services.examples` | Example queries | -| `entities` | `services.entities` | Expected parameters | -| `text_for_embedding` | Computed | Concatenated text for vector embedding | -| `service_id` | `services.service_id` | Link to database record | -| `ruuter_type` | `services.ruuter_type` | HTTP method | -| `current_state` | `services.current_state` | Service status | - -**Embedding Text Construction:** -```python -def construct_embedding_text(service: ServiceRecord) -> str: - """ - Construct text for embedding from service data. - Format: description + examples (newline-separated) - """ - parts = [service.description] - parts.extend(service.examples) - return "\n".join(parts) -``` - -### 7.3 Database → Qdrant Synchronization - -**Trigger Mechanism:** -```sql --- PostgreSQL NOTIFY/LISTEN pattern or polling -CREATE OR REPLACE FUNCTION notify_service_change() -RETURNS TRIGGER AS $$ -BEGIN - IF TG_OP = 'INSERT' OR TG_OP = 'UPDATE' THEN - PERFORM pg_notify( - 'service_sync', - json_build_object( - 'action', TG_OP, - 'service_id', NEW.service_id, - 'current_state', NEW.current_state - )::text - ); - ELSIF TG_OP = 'DELETE' THEN - PERFORM pg_notify( - 'service_sync', - json_build_object( - 'action', 'DELETE', - 'service_id', OLD.service_id - )::text - ); - END IF; - RETURN NEW; -END; -$$ LANGUAGE plpgsql; - -CREATE TRIGGER service_sync_trigger -AFTER INSERT OR UPDATE OR DELETE ON services -FOR EACH ROW EXECUTE FUNCTION notify_service_change(); -``` - -**Sync Service:** -```python -# Location: src/tool_classifier/intent_sync_service.py - -class IntentCollectionSyncService: - """Synchronizes services table with Qdrant intent_collection.""" - - async def handle_service_change(self, event: Dict): - action = event['action'] - service_id = event['service_id'] - - if action in ['INSERT', 'UPDATE']: - # Fetch service from database - service = await self.db.fetch_service(service_id) - - # Generate embedding - embedding_text = self.construct_embedding_text(service) - embedding_vector = await self.embed(embedding_text) - - # Upsert to Qdrant - await self.qdrant_client.upsert( - collection_name="intent_collection", - points=[{ - "id": service.service_id, - "vector": embedding_vector, - "payload": { - "name": service.name, - "description": service.description, - "examples": service.examples, - "entities": service.entities, - "text_for_embedding": embedding_text, - "service_id": service.service_id, - "ruuter_type": service.ruuter_type, - "current_state": service.current_state - } - }] - ) - - elif action == 'DELETE': - await self.qdrant_client.delete( - collection_name="intent_collection", - points_selector={"points": [service_id]} - ) -``` - ---- - -## 8. Error Messages & Constants - -### 8.1 New Error Messages - -**Location:** `src/llm_orchestrator_config/llm_ochestrator_constants.py` - -```python -# Service Workflow Errors -SERVICE_NOT_FOUND_MESSAGES = { - "et": "Vabandust, ma ei leidnud sobivat teenust teie päringu jaoks.", - "en": "Sorry, I couldn't find a matching service for your request.", -} - -SERVICE_VALIDATION_FAILED_MESSAGES = { - "et": "Teenus ei ole hetkel saadaval.", - "en": "The requested service is currently unavailable.", -} - -SERVICE_TIMEOUT_ERROR_MESSAGES = { - "et": "Teenuse vastus võttis liiga kaua aega. Palun proovige hiljem uuesti.", - "en": "The service took too long to respond. Please try again later.", -} - -SERVICE_EXECUTION_ERROR_MESSAGES = { - "et": "Teenuse kutsumine ebaõnnestus. Palun proovige hiljem uuesti.", - "en": "Service execution failed. Please try again later.", -} - -ENTITY_EXTRACTION_FAILED_MESSAGES = { - "et": "Ma ei suutnud teie päringust vajalikku infot tuvastada.", - "en": "I couldn't extract the required information from your query.", -} - -# Context Workflow Errors -INSUFFICIENT_CONTEXT_MESSAGES = { - "et": "Ma ei leia vastust meie eelmisest vestlusest. Kas saate täpsustada?", - "en": "I can't find the answer in our previous conversation. Can you clarify?", -} - -NO_CONTEXT_AVAILABLE_MESSAGES = { - "et": "Mul pole piisavalt konteksti teie küsimusele vastamiseks.", - "en": "I don't have enough context to answer your question.", -} - -# Greeting Responses -GREETING_HELLO_MESSAGES = { - "et": "Tere! Kuidas saan teid aidata?", - "en": "Hello! How can I help you?", -} - -GREETING_GOODBYE_MESSAGES = { - "et": "Head aega! Kui vajate abi, olen siin.", - "en": "Goodbye! If you need help, I'm here.", -} - -GREETING_THANKS_MESSAGES = { - "et": "Pole tänu väärt! Kas saan veel kuidagi aidata?", - "en": "You're welcome! Can I help you with anything else?", -} - -GREETING_CASUAL_MESSAGES = { - "et": "Tere! Mida te soovite teada?", - "en": "Hi there! What would you like to know?", -} -``` - -**Helper Function for Default Greeting Responses:** - -```python -def get_default_greeting_response(greeting_type: str, language: str) -> str: - """ - Get default greeting response based on type and language. - - Args: - greeting_type: Type of greeting ('hello', 'goodbye', 'thanks', 'casual') - language: Language code ('et', 'en') - - Returns: - Localized greeting response - """ - greeting_map = { - "hello": GREETING_HELLO_MESSAGES, - "goodbye": GREETING_GOODBYE_MESSAGES, - "thanks": GREETING_THANKS_MESSAGES, - "casual": GREETING_CASUAL_MESSAGES - } - - messages = greeting_map.get(greeting_type, GREETING_HELLO_MESSAGES) - return messages.get(language, messages["en"]) -``` - -### 8.2 Reused Constants - -```python -# Already defined - reuse for consistency -OUT_OF_SCOPE_MESSAGE -TECHNICAL_ISSUE_MESSAGE -INPUT_GUARDRAIL_VIOLATION_MESSAGE -OUTPUT_GUARDRAIL_VIOLATION_MESSAGE -``` - ---- - -## 9. API Integration - -### 9.1 Entry Points (No Changes) - -The tool classifier is transparent to API consumers. All existing endpoints continue to work: - -**Non-Streaming:** -```http -POST /orchestrate -Content-Type: application/json - -{ - "chatId": "session-123", - "message": "What is the EUR to USD exchange rate?", - "authorId": "user-456", - "conversationHistory": [], - "url": "https://example.com", - "environment": "production", - "connection_id": "conn-789" -} -``` - -**Streaming:** -```http -POST /orchestrate/stream -Content-Type: application/json - -(Same request body as /orchestrate) -``` - -**Testing:** -```http -POST /orchestrate/test -Content-Type: application/json - -{ - "message": "Convert 100 EUR to USD", - "environment": "testing", - "connectionId": 1 -} -``` - -### 9.2 Response Format (No Changes) - -**Success Response:** -```json -{ - "chatId": "session-123", - "llmServiceActive": true, - "questionOutOfLLMScope": false, - "inputGuardFailed": false, - "content": "The current EUR to USD exchange rate is 1.08." -} -``` - -**Service Workflow Response:** -```json -{ - "chatId": "session-123", - "llmServiceActive": true, - "questionOutOfLLMScope": false, - "inputGuardFailed": false, - "content": "Based on the ExchangeRateService: EUR/USD = 1.0850" -} -``` - -The response format remains unchanged. The workflow selection is internal and transparent to the API consumer. - ---- - -## 10. Implementation Considerations - -### 10.1 Performance Optimization - -**Service Discovery Caching:** -```python -# Cache active service count for 5 minutes -@cached(ttl=300) -async def get_active_service_count() -> int: - return await db.count_active_services() -``` - -**Intent Collection Warm-up:** -```python -# Pre-load intent collection on startup -async def warmup_intent_collection(): - """Ensure intent_collection is ready before processing requests.""" - collection_info = await qdrant_client.get_collection("intent_collection") - logger.info(f"Intent collection ready: {collection_info.points_count} services") -``` - -### 10.2 Monitoring & Analytics - -**Tool Classifier Decisions Table:** -```sql --- Track classifier decisions for analytics -CREATE TABLE tool_classifier_decisions ( - id SERIAL PRIMARY KEY, - chat_id TEXT NOT NULL, - author_id TEXT, - user_query TEXT NOT NULL, - detected_workflow VARCHAR(20) NOT NULL, -- 'service', 'context', 'rag', 'ood' - classifier_confidence NUMERIC(5,4), - service_id VARCHAR(100), -- If service workflow - execution_time_ms INTEGER, - created_at TIMESTAMP DEFAULT NOW() -); - -CREATE INDEX idx_classifier_decisions_workflow - ON tool_classifier_decisions(detected_workflow); -``` - -### 10.3 Cost Tracking - -**Add tracking for new LLM calls:** -# Service workflow - intent detection -costs_metric["intent_detection"] = { - "total_prompt_tokens": usage.prompt_tokens, - "total_completion_tokens": usage.completion_tokens, - "total_cost": calculate_cost(usage) -} - -# Context workflow - context availability check -costs_metric["context_check -costs_metric["intent_detection"] = { - "total_prompt_tokens": usage.prompt_tokens, - "total_completion_tokens": usage.completion_tokens, - "total_cost": calculate_cost(usage) -} -``` - -### 10.4 Guardrails Strategy - -**Output Guardrails Application:** -```python -# Apply output guardrails to ALL workflows for consistency -WORKFLOWS_WITH_OUTPUT_GUARDRAILS = [ - WorkflowType.SERVICE, # Check service responses (may contain PII/sensitive data) - WorkflowType.CONTEXT, # Check context-based responses (conversation history may have PII) - WorkflowType.RAG # Existing behavior (knowledge base responses) -] - -# OOD responses skip guardrails (fixed message) -WORKFLOWS_WITHOUT_OUTPUT_GUARDRAILS = [ - WorkflowType.OOD -] -``` - -**Validation-First Approach:** - -All workflows use the **validation-first** approach where content is validated BEFORE streaming to the client: - -1. **RAG Workflow** (existing): - - LLM generates response via streaming - - NeMo buffers tokens (chunk_size=200) - - Each buffer validated before yielding - - Uses `stream_with_guardrails()` method - -2. **Service Workflow** (new): - - External service returns complete response - - Apply output guardrails validation - - Stream validated response token-by-token to client - - Consistent UX with RAG workflow - -3. **Context Workflow** (new): - - LLM returns complete answer from history - - Apply output guardrails validation - - Stream validated response token-by-token to client - - Consistent UX with RAG workflow - -**Streaming + Output Guardrails Integration:** - -```python -# For Service and Context workflows -async def stream_validated_response( - response_text: str, - guardrails_adapter: NeMoRailsAdapter, - request: OrchestrationRequest, - costs_metric: Dict -) -> AsyncIterator[str]: - """ - Apply output guardrails and stream validated response. - - Flow: - 1. Validate complete response with guardrails - 2. If allowed: Stream token-by-token to client - 3. If blocked: Send guardrail violation message - """ - # Check output guardrails (non-streaming validation) - output_check = await guardrails_adapter.check_output_async(response_text) - - # Track costs - costs_metric["output_guardrails"] = output_check.usage - - if not output_check.allowed: - logger.warning(f"[{request.chatId}] Output blocked by guardrails") - # Send violation message - yield format_sse(request.chatId, OUTPUT_GUARDRAIL_VIOLATION_MESSAGE) - yield format_sse(request.chatId, "END") - return - - # Response validated - stream to client - logger.info(f"[{request.chatId}] Streaming validated response") - for token in split_into_tokens(response_text): - yield format_sse(request.chatId, token) - await asyncio.sleep(0.01) # Maintain streaming pace - - yield format_sse(request.chatId, "END") -``` - -**Utility Function for Token Streaming:** -```python -def split_into_tokens(text: str, chunk_size: int = 5) -> List[str]: - """ - Split text into token-like chunks for streaming simulation. - - Used by Service and Context workflows to provide streaming UX - even though the complete response is already available. - - Args: - text: Complete response text - chunk_size: Number of words per chunk - - Returns: - List of text chunks - """ - words = text.split() - tokens = [] - for i in range(0, len(words), chunk_size): - chunk = " ".join(words[i:i + chunk_size]) - tokens.append(chunk + " " if i + chunk_size < len(words) else chunk) - return tokens -``` - -### 10.5 Streaming Implementation Summary - -| Aspect | RAG Workflow | Service Workflow | Context Workflow | -|--------|--------------|------------------|------------------| -| **Response Type** | Streaming (token-by-token) | Complete (all at once) | Complete (all at once) | -| **Validation Timing** | Real-time (buffered chunks) | Pre-validation | Pre-validation | -| **Guardrail Method** | `stream_with_guardrails()` | `check_output_async()` | `check_output_async()` | -| **Streaming Reason** | Natural (LLM streams) | UX consistency | UX consistency | -| **Token Buffering** | NeMo 200-char chunks | Manual 5-word chunks | Manual 5-word chunks | -| **Cost Tracking** | Inline (timing = 0.0) | Separate call | Separate call | -| **Blocked Handling** | Stop mid-stream | Pre-check, don't stream | Pre-check, don't stream | -| **Client Experience** | Progressive reveal | Progressive reveal | Progressive reveal | - -**Implementation Status:** -- RAG streaming + guardrails: **Already implemented** (production-ready) -- Service streaming + guardrails: **To be implemented** (spec complete) -- Context streaming + guardrails: **To be implemented** (spec complete) - ---- - -## 11. Testing Strategy - -### 11.1 Unit Tests -async def test_context_detection_with_llm(): - query = "What did you say earlier?" - history = [ - ConversationItem(authorRole="bot", message="The EUR to USD rate is 1.08"), - ConversationItem(authorRole="user", message="Thanks") - ] - result = await context_analyzer.check_context_availability(query, history) - assert result.can_answer_from_context == True - assert "1.08" in result.answer - -async def test_context_detection_no_reference(): - query = "What are digital signatures?" - history = [ConversationItem(message="The rate is 1.08", ...)] - result = await context_analyzer.check_context_availability(query, history) - assert result.can_answer_from_context == False - -def test_rag_fallback(): - query = "What are digital signatures?" - result = classifier.classify(query, []) - assert result.workflow == WorkflowType.RAG - -async def test_context_streaming(): - """Test that context workflow supports streaming.""" - query = "What was the rate?" - history = [ConversationItem(message="The rate is 1.08", ...)] - - tokens = [] - async for token in context_workflow.execute_streaming(query, history): - tokens.append(token) - - assert len(tokens) > 0 - assert tokens[-1] == "END" - query = "What did you say earlier?" - history = [ConversationItem(message="The rate is 1.08", ...)] - result = classifier.classify(query, history) - assert result.workflow == WorkflowType.CONTEXT - -def test_rag_fallback(): - query = "What are digital signatures?" - result = classifier.classify(query, []) - assert result.workflow == WorkflowType.RAG -``` - -### 11.2 Integration Tests - -```python -# tests/integration_tests/test_service_workflow.py -async def test_full_service_workflow(): - request = OrchestrationRequest( - message="Convert 100 EUR to USD", - chatId="test-123", - ... - ) - response = await orchestration_service.process_orchestration_request(request) - assert response.llmServiceActive == True - assert "exchange rate" in response.content.lower() -``` - -### 11.3 Load `ContextAnalyzer` with LLM-based context checking -- Create context check prompt template with structured output -- Implement `ContextWorkflowExecutor` with streaming support -- Add conversation history formatting utilities -- Integration tests for context workflow (streaming + non-streaming) -- Cost tracking for context check LLM calls>50 services -locust -f tests/load/test_classifier_load.py --users 100 --spawn-rate 10 -``` - ---- - -## 12. Migration Path - -### 12.1 Phase 1: -- Create database migration for `services` table -- Create Qdrant `intent_collection` -- Relocate input guardrails before tool classifier -- Define error message constants - -### 12.2 Phase 2: -- Implement `ToolClassifier` with rule-based logic -- Implement workflow routing in `LLMOrchestrationService` -- Add classifier decision logging -- Unit tests for classifier - -### 12.3 Phase 3: Service Workflow -- Implement `ServiceDiscoveryManager` (Qdrant semantic search) -- Implement `IntentEntityExtractor` (LLM-based) -- Implement `ServiceWorkflowExecutor` (validation & triggering) -- Implement `IntentCollectionSyncService` (DB → Qdrant) -- Integration tests for service workflow - -### 12.4 Phase 4: Context Workflow -- ✅ ImpleHECK_TEMPERATURE=0.0 # Deterministic for classification -CONTEXT_CHECK_MAX_TOKENS=300tection -- Implement conversation history semantic search -- Implement `ContextWorkflowExecutor` -- Integration tests for context workflow - -### 12.5 Phase 5: Finalization -- Extend output guardrails to service & context workflows -- Implement fallback chain (service → context → rag → ood) -- Add comprehensive error handling -- Performance optimization (caching, async) -- End-to-end testing -- Production deployment - ---- - -## 13. Configuration - -### 13.1 Environment Variables - -```bash -# Service Workflow Configuration -RUUTER_BASE_URL=http://ruuter:8086 -SERVICE_DISCOVERY_TIMEOUT=2 # seconds -SERVICE_CALL_TIMEOUT=10 # seconds -MAX_SERVICES_FOR_LLM_CONTEXT=50 - -# Qdrant Configuration -QDRANT_INTENT_COLLECTION=intent_collection -INTENT_SEARCH_TOP_K=20 -INTENT_SEARCH_THRESHOLD=0.5 - -# Context Workflow Configuration -CONTEXT_WINDOW_SIZE=10 -CONTEXT_CONFIDENCE_THRESHOLD=0.7 -``` - -### 13.2 Feature Flags - -```python -# src/llm_orchestrator_config/feature_flags.py - -class FeatureFlags: - # Enable/disable tool classifier (rollback switch) - TOOL_CLASSIFIER_ENABLED = os.getenv("TOOL_CLASSIFIER_ENABLED", "true").lower() == "true" - - # Enable/disable specific workflows - SERVICE_WORKFLOW_ENABLED = os.getenv("SERVICE_WORKFLOW_ENABLED", "true").lower() == "true" - CONTEXT_WORKFLOW_ENABLED = os.getenv("CONTEXT_WORKFLOW_ENABLED", "true").lower() == "true" - - # Fallback to RAG if tool classifier fails - FALLBACK_TO_RAG_ON_ERROR = True -``` - ---- - -## 14. Rollback Strategy - -### 14.1 Graceful Degradation - -```python -def process_orchestration_request(self, request: OrchestrationRequest): - """Process with tool classifier or fallback to RAG.""" - - if not FeatureFlags.TOOL_CLASSIFIER_ENABLED: - # Fallback: Use existing RAG-only pipeline - logger.info("Tool classifier disabled - using RAG pipeline") - return self._execute_rag_workflow(request, None) - - try: - # New: Tool classifier routing - classifier_result = self.tool_classifier.classify(...) - return self._route_to_workflow(request, classifier_result) - - except Exception as e: - logger.error(f"Tool classifier failed: {e}") - if FeatureFlags.FALLBACK_TO_RAG_ON_ERROR: - logger.info("Falling back to RAG workflow") - return self._execute_rag_workflow(request, None) - raise -``` - -## 15. Success Metrics - -### 15.1 Performance Metrics - -| Metric | Target | Measurement | -|--------|--------|-------------| -| Tool Classifier Latency | < 200ms | p95 response time | -| Service Discovery (>50 services) | < 500ms | Qdrant search + LLM intent | -| Service Call Success Rate | > 95% | Successful service executions | -| Context Match Accuracy | > 80% | Correct context-based responses | -| End-to-End Latency | < 3s | Request to response | - -### 15.2 Quality Metrics - -| Metric | Target | Measurement | -|--------|--------|-------------| -| Workflow Classification Accuracy | > 90% | Manual evaluation sample | -| Service Intent Accuracy | > 85% | Correct service selection | -| Entity Extraction Accuracy | > 90% | Correct entity values | -| False Positive Rate (Service) | < 5% | Incorrect service routing | -| User Satisfaction | > 4.0/5.0 | User feedback surveys | - ---- diff --git a/docs/TOOL_CLASSIFIER_SKELETON_USAGE.md b/docs/TOOL_CLASSIFIER_SKELETON_USAGE.md deleted file mode 100644 index 38ce1f53..00000000 --- a/docs/TOOL_CLASSIFIER_SKELETON_USAGE.md +++ /dev/null @@ -1,542 +0,0 @@ -# Tool Classifier Skeleton - Usage Guide - -**Version**: 1.0 -**Date**: February 17, 2026 -**Status**: Skeleton Implementation - ---- - -## Overview - -This skeleton implements the **framework** for a multi-workflow routing system based on the [TOOL_CLASSIFIER_EXTENSION_SPEC.md](./TOOL_CLASSIFIER_EXTENSION_SPEC.md) specification. - -### Current Status - - **Implemented (Skeleton)**: -- Abstract base classes and interfaces -- Workflow executor skeletons (Service, Context, RAG, OOD) -- Tool classifier with classification and routing logic -- Feature flags for safe deployment -- Integration into LLMOrchestrationService - - **Not Implemented (Separate Tasks)**: -- Service discovery logic (Layer 1) -- Context analysis logic (Layer 2) -- Actual LLM calls in workflows -- Output guardrails integration for new workflows -- Database schema changes - -### Current Behavior - -When `TOOL_CLASSIFIER_ENABLED=false` (default): -- System works exactly as before (RAG-only pipeline) -- No changes to existing functionality - -When `TOOL_CLASSIFIER_ENABLED=true`: -- Classifier routes queries (currently always to RAG) -- Service and Context workflows return `None` (fallback to RAG) -- RAG workflow wraps existing pipeline -- All queries ultimately handled by RAG - ---- - -## Architecture - -### Layer-Wise Workflow Routing - -``` -User Query - ↓ -Input Guardrails - ↓ -Tool Classifier - ↓ -┌────────────────┐ -│ Classification │ -└────────┬───────┘ - ↓ - ┌─────┴──────┐ - │ Routing │ - └─────┬──────┘ - ↓ - ╔═══════════════════════════════════╗ - ║ Layer 1: Service Workflow ║ → (returns None - not implemented) - ╚═══════════════════════════════════╝ - ↓ (fallback) - ╔═══════════════════════════════════╗ - ║ Layer 2: Context Workflow ║ → (returns None - not implemented) - ╚═══════════════════════════════════╝ - ↓ (fallback) - ╔═══════════════════════════════════╗ - ║ Layer 3: RAG Workflow ║ → Handles query (existing pipeline) - ╚═══════════════════════════════════╝ - ↓ - Response to User -``` - -### Component Structure - -``` -src/tool_classifier/ -├── __init__.py # Module exports -├── enums.py # WorkflowType enum -├── models.py # ClassificationResult models -├── base_workflow.py # Abstract BaseWorkflow class -├── classifier.py # Main ToolClassifier -└── workflows/ - ├── __init__.py - ├── service_workflow.py # Layer 1 (skeleton) - ├── context_workflow.py # Layer 2 (skeleton) - ├── rag_workflow.py # Layer 3 (complete) - └── ood_workflow.py # Layer 4 (skeleton) -``` - -### Abstract Base Class Pattern - -The system uses **BaseWorkflow** as an abstract base class to ensure all workflows follow the same contract. - -#### How It Works - -1. **BaseWorkflow defines the contract**: - - Every workflow MUST implement two methods: `execute_async()` and `execute_streaming()` - - Both methods return `Optional[...]` to support the fallback pattern (return `None` → next layer) - - Python's `@abstractmethod` decorator enforces this at instantiation time - -2. **All workflows inherit from BaseWorkflow**: - - ServiceWorkflowExecutor extends BaseWorkflow → implements both methods - - ContextWorkflowExecutor extends BaseWorkflow → implements both methods - - RAGWorkflowExecutor extends BaseWorkflow → implements both methods - - OODWorkflowExecutor extends BaseWorkflow → implements both methods - -3. **Classifier treats all workflows uniformly**: - - The `ToolClassifier.route_to_workflow()` method doesn't need to know which specific workflow it's calling - - It just calls `workflow.execute_async()` or `workflow.execute_streaming()` - - This is **polymorphism** - same interface, different behavior - -4. **Benefits**: - - **Consistency**: All workflows have the same interface - - **Enforcement**: Can't create a workflow without implementing required methods - - **Flexibility**: Easy to add new workflows - just extend BaseWorkflow - - **Testability**: Each workflow can be tested independently - - **Fallback Pattern**: `Optional` return type enables layer chaining - -#### Example Flow - -``` -ToolClassifier needs to execute a workflow - ↓ -Gets workflow object (could be Service, Context, RAG, or OOD) - ↓ -Calls workflow.execute_async(request, context) - ↓ -BaseWorkflow contract guarantees this method exists - ↓ -Each workflow implements its own logic - ↓ -Returns OrchestrationResponse or None (fallback to next layer) -``` - -The abstract class is like a **blueprint** that says: "Any workflow in this system MUST be able to do these two things: execute normally and execute with streaming. I don't care *how* you do it, but you must provide these capabilities." - ---- - -## Feature Flags - -### Environment Variables - -```bash -# Master switch (default: false for safe deployment) -TOOL_CLASSIFIER_ENABLED=false - -# Individual workflow toggles (only apply when classifier enabled) -SERVICE_WORKFLOW_ENABLED=true -CONTEXT_WORKFLOW_ENABLED=true -``` - -### Configuration Class - -```python -from src.llm_orchestrator_config.feature_flags import FeatureFlags - -# Check if classifier is enabled -if FeatureFlags.TOOL_CLASSIFIER_ENABLED: - # Use tool classifier - pass - -# Check specific workflow -if FeatureFlags.is_workflow_enabled("service"): - # Service workflow logic - pass - -# Log current configuration -FeatureFlags.log_configuration() -``` - ---- - -## How It Works - -### 1. Non-Streaming Endpoint (`/orchestrate`) - -#### Current Flow (TOOL_CLASSIFIER_ENABLED=false) - -```python -POST /orchestrate - ↓ -LLMOrchestrationService.process_orchestration_request() - ↓ -Initialize components (LLM, guardrails, retriever, generator) - ↓ -Execute RAG pipeline - ↓ -Return OrchestrationResponse -``` - -#### With Classifier (TOOL_CLASSIFIER_ENABLED=true) - -```python -POST /orchestrate - ↓ -LLMOrchestrationService.process_orchestration_request() - ↓ -Initialize components - ↓ -Tool Classifier Integration: - 1. Initialize ToolClassifier (if first time) - 2. Classify query → ClassificationResult - - Currently always returns: WorkflowType.RAG - 3. Route to workflow: - - ServiceWorkflow.execute_async() → returns None - - ContextWorkflow.execute_async() → returns None - - RAGWorkflow.execute_async() → returns response - ↓ -Return OrchestrationResponse -``` - -### 2. Streaming Endpoint (`/orchestrate/stream`) - -#### Current Flow (TOOL_CLASSIFIER_ENABLED=false) - -```python -POST /orchestrate/stream - ↓ -LLMOrchestrationService.stream_orchestration_response() - ↓ -Initialize components - ↓ -Check input guardrails - ↓ -Refine prompt → Retrieve chunks → Stream through NeMo - ↓ -Yield SSE strings -``` - -#### With Classifier (TOOL_CLASSIFIER_ENABLED=true) - -```python -POST /orchestrate/stream - ↓ -LLMOrchestrationService.stream_orchestration_response() - ↓ -Initialize components - ↓ -Check input guardrails - ↓ -Tool Classifier Integration: - 1. Initialize ToolClassifier (if first time) - 2. Classify query → ClassificationResult - 3. Route to streaming workflow: - - ServiceWorkflow.execute_streaming() → returns None - - ContextWorkflow.execute_streaming() → returns None - - RAGWorkflow.execute_streaming() → yields SSE - ↓ -Yield SSE strings -``` - -### 3. Test Endpoint (`/orchestrate/test`) - -Works identically to `/orchestrate`: -- Converts `TestOrchestrationRequest` → `OrchestrationRequest` -- Routes through classifier (if enabled) -- Converts response back to `TestOrchestrationResponse` - ---- - -## Code Examples - -### Using the Classification System - -```python -from src.tool_classifier import ToolClassifier, WorkflowType, ClassificationResult - -# Initialize classifier -classifier = ToolClassifier( - llm_manager=llm_manager, - orchestration_service=service, -) - -# Classify a query -classification = await classifier.classify( - query="Hello, how are you?", - conversation_history=[], - language="en", -) - -# Check result -print(classification.workflow) # WorkflowType.RAG (in skeleton) -print(classification.confidence) # 1.0 -print(classification.reasoning) # "Default to RAG workflow..." - -# Route to workflow -response = await classifier.route_to_workflow( - classification=classification, - request=request, - is_streaming=False, -) -``` - -### Implementing a Workflow (Example) - -```python -from src.tool_classifier.base_workflow import BaseWorkflow -from models.request_models import OrchestrationRequest, OrchestrationResponse - -class MyCustomWorkflow(BaseWorkflow): - """Custom workflow implementation.""" - - async def execute_async( - self, - request: OrchestrationRequest, - context: Dict[str, Any], - ) -> Optional[OrchestrationResponse]: - """Handle query in non-streaming mode.""" - - # Check if this workflow can handle the query - can_handle = await self._check_if_applicable(request.message) - - if not can_handle: - # Return None to trigger fallback to next layer - return None - - # Execute workflow logic - result = await self._process_query(request.message) - - # Validate with output guardrails (TODO) - # is_safe = await guardrails.check_output_async(result) - # if not is_safe: - # return None or violation_response - - # Return response - return OrchestrationResponse( - chatId=request.chatId, - llmServiceActive=True, - questionOutOfLLMScope=False, - inputGuardFailed=False, - content=result, - ) - - async def execute_streaming( - self, - request: OrchestrationRequest, - context: Dict[str, Any], - ) -> Optional[AsyncIterator[str]]: - """Handle query in streaming mode.""" - - # Check if applicable - can_handle = await self._check_if_applicable(request.message) - - if not can_handle: - return None # Fallback - - # Get complete result - result = await self._process_query(request.message) - - # Validate with guardrails (TODO) - # is_safe = await guardrails.check_output_async(result) - # if not is_safe: - # yield format_sse(chatId, VIOLATION_MESSAGE) - # yield format_sse(chatId, "END") - # return - - # Stream result token-by-token - async def stream_result(): - for chunk in self._split_into_tokens(result): - yield self.format_sse(request.chatId, chunk) - await asyncio.sleep(0.01) - yield self.format_sse(request.chatId, "END") - - return stream_result() -``` - ---- - -## Deployment Strategy - -### Phase 1: Testing (Current State) - -```bash -# Keep classifier disabled -TOOL_CLASSIFIER_ENABLED=false -``` - -**Result**: System works exactly as before (RAG-only) - -### Phase 2: Enable Classifier (No Impact) - -```bash -# Enable classifier (but workflows not implemented) -TOOL_CLASSIFIER_ENABLED=true -SERVICE_WORKFLOW_ENABLED=true -CONTEXT_WORKFLOW_ENABLED=true -``` - -**Result**: -- Classifier runs but always routes to RAG -- Service/Context return `None` → fallback to RAG -- Functionally identical to Phase 1 -- Validates integration works - -### Phase 3: Implement Service Workflow - -1. Implement service discovery logic (separate task) -2. Deploy with `SERVICE_WORKFLOW_ENABLED=true` -3. Monitor service routing behavior -4. Rollback flag if issues occur - -### Phase 4: Implement Context Workflow - -1. Implement context analysis logic (separate task) -2. Deploy with `CONTEXT_WORKFLOW_ENABLED=true` -3. Monitor greeting/context detection -4. Rollback flag if issues occur - -### Phase 5: Production - -All workflows operational, full layer-wise routing active. - ---- - -## Extending the System - -### Adding a New Workflow - -1. **Create Workflow Executor**: - -```python -# src/tool_classifier/workflows/custom_workflow.py - -from src.tool_classifier.base_workflow import BaseWorkflow - -class CustomWorkflowExecutor(BaseWorkflow): - """Your custom workflow.""" - - async def execute_async(self, request, context): - # Implement logic - pass - - async def execute_streaming(self, request, context): - # Implement streaming logic - pass -``` - -2. **Register in Classifier**: - -```python -# src/tool_classifier/enums.py - -class WorkflowType(Enum): - SERVICE = "service" - CONTEXT = "context" - RAG = "rag" - CUSTOM = "custom" # Add new type - OOD = "ood" - -# Update layer order -WORKFLOW_LAYER_ORDER = [ - WorkflowType.SERVICE, - WorkflowType.CONTEXT, - WorkflowType.CUSTOM, # Add to chain - WorkflowType.RAG, - WorkflowType.OOD, -] -``` - -3. **Initialize in ToolClassifier**: - -```python -# src/tool_classifier/classifier.py - -def __init__(self, ...): - # ... existing workflows ... - self.custom_workflow = CustomWorkflowExecutor(...) -``` - -4. **Add Feature Flag**: - -```python -# src/llm_orchestrator_config/feature_flags.py - -CUSTOM_WORKFLOW_ENABLED = ( - os.getenv("CUSTOM_WORKFLOW_ENABLED", "true").lower() == "true" -) -``` - ---- - -## Key Concepts - -### 1. None Return Pattern - -Workflows return `None` when they cannot handle a query: - -```python -if not can_handle: - return None # Triggers fallback to next layer -``` - -This enables the fallback chain: Service → Context → RAG → OOD - -### 2. Validation-First Streaming - -For Service and Context workflows (complete responses): - -```python -# 1. Get complete response -response = await call_service(...) - -# 2. Validate BEFORE streaming -is_safe = await guardrails.check_output_async(response) - -if not is_safe: - yield format_sse(chatId, VIOLATION_MESSAGE) - yield format_sse(chatId, "END") - return - -# 3. Stream validated response -for chunk in split_into_tokens(response): - yield format_sse(chatId, chunk) -yield format_sse(chatId, "END") -``` - -### 3. Two Execution Methods - -Every workflow implements both: -- `execute_async()` → For `/orchestrate` (returns complete response) -- `execute_streaming()` → For `/orchestrate/stream` (yields SSE strings) - ---- - -## Summary - -This skeleton provides: - - **Complete framework** for multi-workflow routing - **Safe deployment** with feature flags - **Extensible architecture** using OOP patterns - **Backward compatibility** (disabled by default) - **Clear contracts** via abstract base classes - **Documentation** for implementation tasks - -The system is ready for workflow implementation in separate, independent tasks. - ---- diff --git a/docs/images/LLM Module App Diagram (Current).png b/docs/images/LLM Module App Diagram (Current).png new file mode 100644 index 00000000..12a44379 Binary files /dev/null and b/docs/images/LLM Module App Diagram (Current).png differ diff --git a/docs/images/LLM Module Context Diagram (Current).png b/docs/images/LLM Module Context Diagram (Current).png new file mode 100644 index 00000000..4e4b7ca7 Binary files /dev/null and b/docs/images/LLM Module Context Diagram (Current).png differ diff --git a/docs/images/LLM Orchestration Service Component Diagram (Current).png b/docs/images/LLM Orchestration Service Component Diagram (Current).png new file mode 100644 index 00000000..d011b9e5 Binary files /dev/null and b/docs/images/LLM Orchestration Service Component Diagram (Current).png differ diff --git a/endpoints.md b/endpoints.md deleted file mode 100644 index 262e81a3..00000000 --- a/endpoints.md +++ /dev/null @@ -1,683 +0,0 @@ -# LLM Connections API Endpoints - -## Base URL -``` -/ruuter-private/llm/connections -``` - ---- - -## 1. Create LLM Connection - -### Endpoint -```http -POST /ruuter-private/llm/connections/create -``` - -### Request Body -```json -{ - "llmPlatform": "OpenAI", - "llmModel": "GPT-4o", - "embeddingPlatform": "OpenAI", - "embeddingModel": "text-embedding-3-small", - "monthlyBudget": 1000.00, - "deploymentEnvironment": "Testing", - // Azure credentials (optional) - "deploymentName": "my-deployment", - "targetUri": "https://my-endpoint.azure.com", - "apiKey": "azure-api-key", - // AWS Bedrock credentials (optional) - "secretKey": "aws-secret-key", - "accessKey": "aws-access-key", - // Embedding model credentials (optional) - "embeddingModelApiKey": "embedding-api-key" -} -``` - -### Response (201 Created) -```json -{ - "id": 1, - "llmPlatform": "OpenAI", - "llmModel": "GPT-4o", - "embeddingPlatform": "OpenAI", - "embeddingModel": "text-embedding-3-small", - "monthlyBudget": 1000.00, - "usedBudget": 0.00, - "deploymentEnvironment": "Testing", - "status": "active", - "createdAt": "2025-09-02T10:15:30.000Z", - // Azure credentials (if provided) - "deploymentName": "my-deployment", - "targetUri": "https://my-endpoint.azure.com", - "apiKey": "azure-api-key", - // AWS Bedrock credentials (if provided) - "secretKey": "aws-secret-key", - "accessKey": "aws-access-key", - // Embedding model credentials (if provided) - "embeddingModelApiKey": "embedding-api-key" -} -``` - ---- - -## 2. Update LLM Connection - -### Endpoint -```http -POST /ruuter-private/llm/connections/update -``` - -### Request Body -```json -{ - "connectionId": 1, - "llmPlatform": "Azure AI", - "llmModel": "GPT-4o-mini", - "embeddingPlatform": "Azure AI", - "embeddingModel": "text-embedding-ada-002", - "monthlyBudget": 2000.00, - "deploymentEnvironment": "Production", - // Azure credentials (optional) - "deploymentName": "updated-deployment", - "targetUri": "https://updated-endpoint.azure.com", - "apiKey": "updated-azure-api-key", - // AWS Bedrock credentials (optional) - "secretKey": "updated-aws-secret-key", - "accessKey": "updated-aws-access-key", - // Embedding model credentials (optional) - "embeddingModelApiKey": "updated-embedding-api-key" -} -``` - -### Response (200 OK) -```json -{ - "id": 1, - "llmPlatform": "Azure AI", - "llmModel": "GPT-4o-mini", - "embeddingPlatform": "Azure AI", - "embeddingModel": "text-embedding-ada-002", - "monthlyBudget": 2000.00, - "usedBudget": 150.75, - "deploymentEnvironment": "Production", - "status": "active", - "createdAt": "2025-09-02T10:15:30.000Z", - // Azure credentials (if provided) - "deploymentName": "updated-deployment", - "targetUri": "https://updated-endpoint.azure.com", - "apiKey": "updated-azure-api-key", - // AWS Bedrock credentials (if provided) - "secretKey": "updated-aws-secret-key", - "accessKey": "updated-aws-access-key", - // Embedding model credentials (if provided) - "embeddingModelApiKey": "updated-embedding-api-key" -} -``` - ---- - -## 3. Get LLM Connections (Paginated List) - -### Endpoint -```http -POST /ruuter-private/rag-search/llm-connections/list -``` - -### Request Body -```json -{ - "page": 1, - "page_size": 10, - "sorting": "created_at desc" -} -``` - -### Request Parameters -| Parameter | Type | Required | Description | Default | -|-----------|------|----------|-------------|---------| -| `page` | number | No | Page number (1-based) | 1 | -| `page_size` | number | No | Number of items per page | 10 | -| `sorting` | string | No | Sorting criteria | "created_at desc" | - -### Sorting Options -- `llm_platform asc/desc` -- `llm_model asc/desc` -- `embedding_platform asc/desc` -- `embedding_model asc/desc` -- `monthly_budget asc/desc` -- `environment asc/desc` -- `status asc/desc` -- `created_at asc/desc` -- `updated_at asc/desc` - -### Response (200 OK) -```json -[ - { - "id": 1, - "llmPlatform": "OpenAI", - "llmModel": "GPT-4o", - "embeddingPlatform": "OpenAI", - "embeddingModel": "text-embedding-3-small", - "monthlyBudget": 1000.00, - "environment": "Testing", - "status": "active", - "createdAt": "2025-09-02T10:15:30.000Z", - "updatedAt": "2025-09-02T10:15:30.000Z", - "totalPages": 3 - }, - { - "id": 2, - "llmPlatform": "Azure AI", - "llmModel": "GPT-4o-mini", - "embeddingPlatform": "Azure AI", - "embeddingModel": "Ada-200-1", - "monthlyBudget": 2000.00, - "environment": "Production", - "status": "active", - "createdAt": "2025-09-02T09:30:15.000Z", - "updatedAt": "2025-09-02T11:00:00.000Z", - "totalPages": 3 - } -] -``` - ---- - -## 4. Get Single LLM Connection - -### Endpoint -```http -POST /ruuter-private/rag-search/llm-connections/get -``` - -### Request Body -```json -{ - "connection_id": 1 -} -``` - -### Response (200 OK) -```json -{ - "id": 1, - "llmPlatform": "OpenAI", - "llmModel": "GPT-4o", - "embeddingPlatform": "OpenAI", - "embeddingModel": "text-embedding-3-small", - "monthlyBudget": 1000.00, - "environment": "Testing", - "status": "active", - "createdAt": "2025-09-02T10:15:30.000Z", - "updatedAt": "2025-09-02T10:15:30.000Z" -} -``` - -### Response (404 Not Found) -```json -"error: connection not found" -``` - ---- - -## 5. Add New LLM Connection - -### Endpoint -```http -POST /ruuter-private/rag-search/llm-connections/add -``` - -### Request Body -```json -{ - "llm_platform": "OpenAI", - "llm_model": "GPT-4o", - "embedding_platform": "OpenAI", - "embedding_model": "text-embedding-3-small", - "monthly_budget": 1000.00, - "environment": "Testing" -} -``` - -### Request Parameters -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `llm_platform` | string | Yes | LLM platform (e.g., "Azure AI", "OpenAI") | -| `llm_model` | string | Yes | LLM model (e.g., "GPT-4o") | -| `embedding_platform` | string | Yes | Embedding platform | -| `embedding_model` | string | Yes | Embedding model | -| `monthly_budget` | number | Yes | Monthly budget amount | -| `environment` | string | Yes | "Testing" or "Production" | - -### Response (200 OK) -```json -{ - "id": 3, - "llm_platform": "OpenAI", - "llm_model": "GPT-4o", - "embedding_platform": "OpenAI", - "embedding_model": "text-embedding-3-small", - "monthly_budget": 1000.00, - "environment": "Testing", - "status": "active", - "created_at": "2025-09-02T12:00:00.000Z", - "updated_at": "2025-09-02T12:00:00.000Z" -} -``` - -### Response (400 Bad Request) -```json -"error: environment must be 'Testing' or 'Production'" -``` - ---- - -## 6. Update LLM Connection - -### Endpoint -```http -POST /ruuter-private/rag-search/llm-connections/edit -``` - -### Request Body -```json -{ - "connection_id": 1, - "llm_platform": "Azure AI", - "llm_model": "GPT-4o-mini", - "embedding_platform": "Azure AI", - "embedding_model": "Ada-200-1", - "monthly_budget": 2000.00, - "environment": "Production" -} -``` - -### Response (200 OK) -```json -{ - "id": 1, - "llm_platform": "Azure AI", - "llm_model": "GPT-4o-mini", - "embedding_platform": "Azure AI", - "embedding_model": "Ada-200-1", - "monthly_budget": 2000.00, - "environment": "Production", - "status": "active", - "created_at": "2025-09-02T10:15:30.000Z", - "updated_at": "2025-09-02T12:30:00.000Z" -} -``` - -### Response (404 Not Found) -```json -"error: connection not found" -``` - ---- - -## 7. Delete LLM Connection - -### Endpoint -```http -POST /ruuter-private/rag-search/llm-connections/delete -``` - -### Request Body -```json -{ - "connection_id": 1 -} -``` - -### Response (200 OK) -```json -"LLM connection deleted successfully" -``` - -### Response (404 Not Found) -```json -"error: connection not found" -``` - ---- - -## 4. List All LLM Connections - -### Endpoint -```http -GET /ruuter-private/llm/connections/list -``` - -### Query Parameters (Optional for filtering) -| Parameter | Type | Description | -|-----------|------|-------------| -| `llmPlatform` | `string` | Filter by LLM platform | -| `llmModel` | `string` | Filter by LLM model | -| `deploymentEnvironment` | `string` | Filter by environment (Testing / Production) | -| `pageNumber` | `number` | Page number (1-based) | -| `pageSize` | `number` | Number of items per page | -| `sortBy` | `string` | Field to sort by | -| `sortOrder` | `string` | Sort order: 'asc' or 'desc' | - -### Example Request -```http -GET /ruuter-private/llm/connections/list?llmPlatform=OpenAI&deploymentEnvironment=Testing&model=GPT4 -``` - ---- - -## 5. Get Production LLM Connection (with filters) - -### Endpoint -```http -GET /ruuter-private/llm/connections/production -``` - -### Query Parameters (Optional for filtering) -| Parameter | Type | Description | -|-----------|------|-------------| -| `llmPlatform` | `string` | Filter by LLM platform | -| `llmModel` | `string` | Filter by LLM model | -| `embeddingPlatform` | `string` | Filter by embedding platform | -| `embeddingModel` | `string` | Filter by embedding model | -| `connectionStatus` | `string` | Filter by connection status | -| `sortBy` | `string` | Field to sort by | -| `sortOrder` | `string` | Sort order: 'asc' or 'desc' | - -### Example Request -```http -GET /ruuter-private/llm/connections/production?llmPlatform=OpenAI&connectionStatus=active -``` - -### Response (200 OK) -```json -[ - { - "id": 1, - "llmPlatform": "OpenAI", - "llmModel": "GPT-4o", - "embeddingPlatform": "OpenAI", - "embeddingModel": "text-embedding-3-small", - "monthlyBudget": 1000.00, - "deploymentEnvironment": "Testing", - "status": "active", - "createdAt": "2025-09-02T10:15:30.000Z", - "updatedAt": "2025-09-02T10:15:30.000Z" - } -] -``` - ---- - -## 5. Get Single LLM Connection - -### Endpoint -```http -GET /ruuter-private/llm/connections/overview -``` - -### Response (200 OK) -```json -{ - "id": 1, - "llmPlatform": "OpenAI", - "llmModel": "GPT-4o", - "embeddingPlatform": "OpenAI", - "embeddingModel": "text-embedding-3-small", - "monthlyBudget": 1000.00, - "deploymentEnvironment": "Testing", - "status": "active", - "createdAt": "2025-09-02T10:15:30.000Z", - "updatedAt": "2025-09-02T10:15:30.000Z" -} -``` - ---- -# Inference Results API Endpoints - -## Base URL -``` -/ruuter-private/inference/results -``` - ---- - -## 1. Store Test Inference Result - -### Endpoint -```http -POST /ruuter-private/inference/results/test/store -``` - -### Request Body -```json -{ - "llm_connection_id": 1, - "user_question": "What are the benefits of using LLMs?", - "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." -} -``` - -### Request Parameters -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `llm_connection_id` | number | Yes | ID of the LLM connection | -| `user_question` | string | Yes | User's raw question/input | -| `final_answer` | string | Yes | LLM's final generated answer | - -### Response (200 OK) -```json -{ - "data": { - "id": 10, - "llm_connection_id": 1, - "chat_id": null, - "user_question": "What are the benefits of using LLMs?", - "refined_questions": null, - "conversation_history": null, - "ranked_chunks": null, - "embedding_scores": null, - "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", - "environment": "testing", - "created_at": "2025-09-25T12:15:00.000Z" - }, - "operationSuccess": true, - "statusCode": 200 -} -``` - -### Response (400 Bad Request) -```json -{ - "data": "[]", - "operationSuccess": false, - "statusCode": 400 -} -``` - -### Response (404 Not Found) -```json -"error: LLM connection not found" -``` - ---- - -## 2. Store Production Inference Result - -### Endpoint -```http -POST /ruuter-private/inference/results/production/store -``` - -### Request Body -```json -{ - "chat_id": "chat-12345", - "user_question": "What are the benefits of using LLMs?", - "refined_questions": [ - "How do LLMs improve productivity?", - "What are practical use cases of LLMs?" - ], - "conversation_history": [ - { "role": "user", "content": "Hello" }, - { "role": "assistant", "content": "Hi! How can I help you?" } - ], - "ranked_chunks": [ - { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, - { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } - ], - "embedding_scores": [0.92, 0.85, 0.78], - "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." -} -``` - -### Request Parameters -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `chat_id` | string | No | Optional chat session ID | -| `user_question` | string | Yes | User's raw question/input | -| `refined_questions` | object | No | List of refined questions (LLM-generated) | -| `conversation_history` | object | No | Prior messages array of {role, content} | -| `ranked_chunks` | object | No | Retrieved chunks ranked with metadata | -| `embedding_scores` | object | No | Distance scores for each chunk | -| `final_answer` | string | Yes | LLM's final generated answer | - -### Response (200 OK) -```json -{ - "data": { - "id": 15, - "llm_connection_id": null, - "chat_id": "chat-12345", - "user_question": "What are the benefits of using LLMs?", - "refined_questions": [ - "How do LLMs improve productivity?", - "What are practical use cases of LLMs?" - ], - "conversation_history": [ - { "role": "user", "content": "Hello" }, - { "role": "assistant", "content": "Hi! How can I help you?" } - ], - "ranked_chunks": [ - { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, - { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } - ], - "embedding_scores": [0.92, 0.85, 0.78], - "final_answer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", - "environment": "production", - "created_at": "2025-09-25T12:15:00.000Z" - }, - "operationSuccess": true, - "statusCode": 200 -} -``` - -### Response (400 Bad Request) -```json -{ - "data": "[]", - "operationSuccess": false, - "statusCode": 400 -} -``` - ---- - -## 3. View/get Inference Result - -### Endpoint -```http -POST /ruuter-private/inference/results/test/store -``` - -### Request Body -```json -{ - "llmConnectionId": 1, - "userQuestion": "What are the benefits of using LLMs?", - "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." -} -``` - -### Response (201 Created) -```json -{ - "data": { - "id": 15, - "llmConnectionId": 1, - "userQuestion": "What are the benefits of using LLMs?", - "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", - "environment": "testing", - "createdAt": "2025-09-25T10:15:30.000Z" - }, - "operationSuccess": true, - "statusCode": 200 -} -``` - -## 4. Inquiry from chatbot to llm orchestration service - -### Endpoint -```http -POST /ruuter-private/inference/results/production/store -``` - -### Request Body -```json -{ - "llmConnectionId": 1, - "chatId": "chat-session-12345", - "userQuestion": "What are the benefits of using LLMs?", - "refinedQuestions": [ - "How do LLMs improve productivity?", - "What are practical use cases of LLMs?" - ], - "conversationHistory": [ - { "role": "user", "content": "Hello" }, - { "role": "assistant", "content": "Hi! How can I help you?" } - ], - "rankedChunks": [ - { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, - { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } - ], - "embeddingScores": { - "chunk_1": 0.92, - "chunk_2": 0.85 - }, - "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation." -} -``` - -### Response (201 Created) -```json -{ - "id": 20, - "llmConnectionId": 1, - "chatId": "chat-session-12345", - "userQuestion": "What are the benefits of using LLMs?", - "refinedQuestions": [ - "How do LLMs improve productivity?", - "What are practical use cases of LLMs?" - ], - "conversationHistory": [ - { "role": "user", "content": "Hello" }, - { "role": "assistant", "content": "Hi! How can I help you?" } - ], - "rankedChunks": [ - { "id": "chunk_1", "content": "LLMs help in summarization", "rank": 1 }, - { "id": "chunk_2", "content": "They improve Q&A systems", "rank": 2 } - ], - "embeddingScores": { - "chunk_1": 0.92, - "chunk_2": 0.85 - }, - "finalAnswer": "LLMs can improve productivity by summarizing large documents, enabling Q&A, and enhancing automation.", - "environment": "production", - "createdAt": "2025-09-25T10:15:30.000Z" -} -``` - ---- \ No newline at end of file