complete API semantic searcher with ambiguous result handling and tool classifier routing

Author: nuwangeek

DSL/Ruuter.public/rag-search/POST/api-tools/search.yml

@@ -0,0 +1,81 @@
+declaration:
+  call: declare
+  version: 0.1
+  description: "Search API tool endpoints using semantic (hybrid) search against api_tool_collection(test endpoint)"
+  method: post
+  accepts: json
+  returns: json
+  namespace: rag-search
+  allowlist:
+    body:
+      - field: query
+        type: string
+        description: "Natural-language user query to search API endpoints"
+      - field: top_k
+        type: integer
+        description: "Max number of results to return (default: 5)"
+      - field: environment
+        type: string
+        description: "Embedding environment (default: production)"
+
+extract_request_data:
+  assign:
+    query: ${incoming.body.query}
+    top_k: ${incoming.body.top_k || 5}
+    environment: ${incoming.body.environment || 'production'}
+  next: validate_query
+
+validate_query:
+  switch:
+    - condition: "${!query || query.trim() === ''}"
+      next: return_missing_query
+  next: execute_search
+
+return_missing_query:
+  assign:
+    error_data:
+      success: false
+      error: "MISSING_QUERY"
+      message: "'query' field is required and must be a non-empty string"
+  next: return_bad_request
+
+execute_search:
+  call: http.post
+  args:
+    url: "[#RAG_SEARCH_LLM_SERVICE]/api-tools/search"
+    body:
+      query: ${query}
+      top_k: ${top_k}
+      environment: ${environment}
+  result: search_result
+  on_error: handle_search_error
+  next: check_search_status
+
+check_search_status:
+  switch:
+    - condition: ${200 <= search_result.response.statusCodeValue && search_result.response.statusCodeValue < 300}
+      next: return_ok
+  next: handle_search_error
+
+handle_search_error:
+  assign:
+    error_data:
+      success: false
+      error: "SEARCH_FAILED"
+      message: "Semantic search failed. LLM service may be unavailable."
+  next: return_server_error
+
+return_ok:
+  status: 200
+  return: ${search_result.response.body}
+  next: end
+
+return_bad_request:
+  status: 400
+  return: ${error_data}
+  next: end
+
+return_server_error:
+  status: 500
+  return: ${error_data}
+  next: end

constants.ini

@@ -12,4 +12,5 @@ DOMAIN=localhost
 DB_PASSWORD=dbadmin
 RAG_SEARCH_RUUTER_PUBLIC_INTERNAL_SERVICE=http://ruuter:8086/services
 SERVICE_DMAPPER_HBS=http://data-mapper:3000/hbs/rag-search
-SERVICE_PROJECT_LAYER=services
+SERVICE_PROJECT_LAYER=services
+RAG_SEARCH_LLM_SERVICE=http://llm-orchestration-service:8100

docs/API_TOOL_CALLING.md

@@ -8,12 +8,12 @@ API Tool Calling enables the LLM module to discover and invoke external API endp
 in response to user queries. endpoints are
 registered, semantically indexed in Qdrant, and retrieved at query time using hybrid search.
 
-The feature has two halves:
 
-| Half | What it does | Status |
+| component | What it does | Status |
 |---|---|---|
-| **Indexing pipeline** | Takes an endpoint definition → enriches it with LLM context → stores hybrid vectors in Qdrant |  Complete |
-| **Tool classifier** | At query time, routes to the best matching endpoint via hybrid search |  In progress |
+| **Indexing pipeline** | Takes an endpoint definition → enriches it with LLM context → stores hybrid vectors in Qdrant | ✅ Complete |
+| **Tool classifier** | At query time, routes to the best matching endpoint via hybrid search + LLM disambiguation | ✅ Complete |
+| **Workflow executor** | Surfaces the matched endpoint; full agentic loop (param collection → API call) planned | 🔧 Partial (Task 10) |
 
 ---
 
@@ -30,7 +30,11 @@ main_indexer.py  (indexing pipeline)
         ↓  upsert
 api_tool_collection  (Qdrant)
         ↑  query at runtime
-ToolClassifier  (src/tool_classifier/)
+APISemanticSearcher  (src/tool_classifier/api_semantic_searcher.py)
+        ↑  called by
+ToolClassifier._try_api_tool_classification()
+        ↓  ClassificationResult(workflow=API_TOOL_CALLING)
+APIToolWorkflowExecutor  (src/tool_classifier/workflows/api_tool_workflow.py)
 ```
 
 ---
@@ -97,7 +101,7 @@ Defined in [DSL/CronManager/script/api_tool_indexer.sh](../DSL/CronManager/scrip
 
 **What it does (in order):**
 
-1. Validates required env vars (`endpoint_id`, `name`, `description`)
+1. Validates required env vars (`endpoint_id`, `name`, `description`, `url`)
 2. Activates the pre-built Python venv at `/app/python_virtual_env`
 3. Installs required packages via `uv pip install` (`httpx`, `pydantic`, `qdrant-client`, `loguru`)
 4. Sets `PYTHONPATH` to include `/app/src`
@@ -235,4 +239,192 @@ loop can execute the API call without an additional database round-trip.
 | `required` | bool | Whether the caller must supply this param |
 | `description` | str | Human-readable description |
 
+---
+
+
+## Part 2 — Tool Classifier (Query-Time)
+
+### Overview
+
+At query time, `ToolClassifier` in [src/tool_classifier/classifier.py](../src/tool_classifier/classifier.py) the layer by layer execution happens 
+
+
+1. **Service search** → `intent_collections` (Qdrant) — existing Bürokratt services
+2. **API Tool search** → `api_tool_collection` (Qdrant) — registered API tool endpoints
+
+API tool search (`_try_api_tool_classification`) is triggered when:
+- `SERVICE_WORKFLOW_ENABLED=false` (service workflow disabled globally)
+- Dense service search returns no results
+- Service cosine score falls below `DENSE_MIN_THRESHOLD`
+
+It is **always** tried before falling back to Context/RAG.
+
+---
+
+### Component: `APISemanticSearcher`
+
+Defined in [src/tool_classifier/api_semantic_searcher.py](../src/tool_classifier/api_semantic_searcher.py).
+
+Instantiated once in `ToolClassifier.__init__()` and reuses the shared Qdrant `httpx.AsyncClient`.
+
+**Constructor:**
+
+```python
+APISemanticSearcher(
+    embedding_service=orchestration_service,   # generates dense embeddings
+    qdrant_client=self._qdrant_client,         # shared connection pool
+    disambiguator=None,                        # optional: inject for testing
+)
+```
+
+**Key constants** (from `constants.py`):
+
+| Constant | Value | Purpose |
+|---|---|---|
+| `API_TOOL_COLLECTION` | `api_tool_collection` | Qdrant collection name |
+| `API_TOOL_SEARCH_TOP_K` | `5` | Max hybrid results |
+| `API_TOOL_MIN_THRESHOLD` | cosine threshold | Below this → no match |
+| `API_TOOL_HIGH_CONFIDENCE_THRESHOLD` | cosine threshold | Above this → high confidence |
+| `API_TOOL_SCORE_GAP_THRESHOLD` | gap threshold | Minimum lead over runner-up |
+
+---
+
+### Search Flow: `APISemanticSearcher.search()`
+
+```
+User query
+    │
+    ├─ precomputed_embedding provided? → reuse it (no extra API call)
+    └─ otherwise → generate dense embedding via embedding_service
+    │
+    ▼
+Step 1: Dense search (api_tool_collection)
+    → Real cosine similarity scores per endpoint
+    │
+    ├─ No results → return []
+    ├─ top_cosine < API_TOOL_MIN_THRESHOLD → return []
+    └─ continue
+    │
+    ▼
+Step 2: Hybrid search (dense + sparse/BM25 + RRF)
+    → Best-ranked results by RRF fusion score
+    │  Falls back to dense results if hybrid returns nothing
+    │
+    ▼
+Step 3: Annotate confidence for each hybrid result
+    │
+    │  cosine lookup: dense_cosine_map[endpoint_id]
+    │             └─ fallback: point["cosine_score"] (sparse-driven result)
+    │             └─ skip if neither available
+    │
+    │  effective_gap = this_cosine − best_other_cosine_in_dense
+    │
+    ├─ i==0 AND cosine ≥ HIGH_THRESHOLD AND effective_gap ≥ GAP_THRESHOLD → "high"
+    ├─ cosine ≥ MIN_THRESHOLD → "medium"
+    └─ else → skip
+    │
+    ▼
+Step 4: Resolve to exactly one result
+    ├─ high-confidence result exists → return immediately
+    ├─ single medium + large gap → return directly
+    └─ multiple medium OR small gap → LLM disambiguation
+           │
+           └─ EndpointDisambiguatorModule (DSPy + asyncio.to_thread)
+                  → picks winner or returns None
+                  → None means no match → return []
+```
+
+---
+
+### Embedding Reuse
+
+When `ToolClassifier.classify()` already generated a dense embedding for the service
+search, it passes it as `precomputed_embedding` to `_try_api_tool_classification`:
+
+```python
+api_tool_result = await self._try_api_tool_classification(
+    query, request, precomputed_embedding=query_embedding
+)
+```
+
+`APISemanticSearcher.search()` skips the embedding step entirely when this is provided,
+saving one embedding API call per request.
+
+---
+
+### LLM Disambiguation: `EndpointDisambiguatorModule`
+
+Used when multiple medium-confidence endpoints score similarly and no clear winner
+can be determined from cosine scores alone.
+
+- DSPy `Predict` module with `EndpointDisambiguationSignature`
+- Inputs: `user_query` + `candidates` (JSON list of `{endpoint_id, name, description, cosine_score}`)
+- Output: `best_endpoint_id` — the winning `endpoint_id`, or `"none"` if no match
+- Run via `asyncio.to_thread()` to avoid blocking the async event loop
+- Understands Estonian, Russian, and English queries
+
+---
+
+### Feature Flag
+
+API tool calling is gated by `FeatureFlags.API_TOOL_CALLING_WORKFLOW_ENABLED`.
+When `false`, `_try_api_tool_classification` returns `None` immediately without
+touching Qdrant.
+
+---
+
+### Component: `APIToolWorkflowExecutor`
+
+Defined in [src/tool_classifier/workflows/api_tool_workflow.py](../src/tool_classifier/workflows/api_tool_workflow.py).
+
+Handles `WorkflowType.API_TOOL_CALLING` after `ToolClassifier.classify()` has set
+`matched_endpoint` in the context dict.
+
+**Current behaviour (Task 4.1):**
+
+Reads `context["matched_endpoint"]` and returns a simple confirmation response:
+
+```
+**{name}**: {description}
+
+URL: {url}
+```
+
+**Planned (Task 10):** Full agentic loop —
+session management → parameter collection dialog → external API call → response formatting.
+
+---
+
+### End-to-End Flow (Query Time)
+
+```
+User: "What are the public holidays in Estonia?"
+    │
+    ▼
+ToolClassifier.classify()
+    │
+    ├─ Dense search (intent_collections) → low cosine → below threshold
+    │
+    └─ _try_api_tool_classification()
+           │
+           └─ APISemanticSearcher.search()
+                  │
+                  ├─ Dense: get_national_holidays cosine=0.87
+                  ├─ Hybrid: get_national_holidays ranked #1 (RRF)
+                  ├─ effective_gap large → confidence="high"
+                  └─ return [APIToolSearchResult(name="get_national_holidays", ...)]
+           │
+           └─ ClassificationResult(
+                  workflow=API_TOOL_CALLING,
+                  metadata={"matched_endpoint": {...}}
+              )
+    │
+    ▼
+ToolClassifier._execute_with_fallback_async()
+    │
+    └─ APIToolWorkflowExecutor.execute_async(context={"matched_endpoint": {...}})
+           │
+           └─ OrchestrationResponse(content="**get_national_holidays**: ...")
+```
+
 ---

src/api_tool_indexer/constants.py

@@ -31,7 +31,9 @@ class ApiToolIndexerConstants:
     REQUEST_TIMEOUT = 60  # seconds
 
     # Context Enrichment Template
-    # Used to generate a rich semantic context for each endpoint before embedding
+    # Mirrors the service workflow (intent_data_enrichment/constants.py).
+    # Full template goes in chunk_prompt; document_prompt is left empty.
+    # The LLM summarises the chunk content into a rich semantic context.
     CONTEXT_TEMPLATE = """<document>
 {full_endpoint_info}
 </document>
@@ -44,12 +46,24 @@ class ApiToolIndexerConstants:
 </endpoint>
 
 Please generate a rich, detailed context that describes this API endpoint comprehensively for semantic search.
-Include information about:
+Keep the prose context general and country-agnostic. Include information about:
 - What the user wants to accomplish by calling this endpoint
 - Key terms and synonyms for this action
 - Related concepts and use cases
 - Common ways users might ask for this functionality in natural language
 
-IMPORTANT: Generate the context in the SAME LANGUAGE as the endpoint description above. If the description is in Estonian, respond in Estonian. If in English, respond in English. If in Russian, respond in Russian.
+Then, on a new line, add a section exactly as shown below with 6 to 8 realistic and diverse example questions a real user might ask when they need this endpoint. Cover different phrasings, synonyms, and indirect ways of asking — do not just repeat the description verbatim.
 
-Answer only with the enriched context and nothing else."""
+IMPORTANT for example queries: This is a system built for Estonian government digital services (Bürokratt). Ground the examples in an Estonian context — use Estonian cities (Tallinn, Tartu, Pärnu, Narva), Estonian institutions, and Estonia-relevant scenarios. Only use non-Estonian locations if the endpoint is explicitly about comparing or fetching data for multiple countries.
+
+Example queries:
+- <example question 1>
+- <example question 2>
+- <example question 3>
+- <example question 4>
+- <example question 5>
+- <example question 6>
+
+IMPORTANT: Generate everything in the SAME LANGUAGE as the endpoint description above. If the description is in Estonian, respond in Estonian. If in English, respond in English. If in Russian, respond in Russian.
+
+Answer only with the enriched context and example queries — nothing else."""