Semantic searcher evaluation and update to multi point indexing strategy (buerokratt#419)

* remove unwanted file

* updated changes

* fixed requested changes

* fixed issue

* service workflow implementation without calling service endpoints

* fixed requested changes

* fixed issues

* protocol related requested changes

* fixed requested changes

* update time tracking

* added time tracking and reloacate input guardrail before toolclassifiier

* fixed issue

* fixed issue

* added hybrid search for the service detection

* update tool classifier

* fixing merge conflicts

* fixed issue

* optimize first user query response generation time

* fixed pr reviewed issues

* service integration

* context based response generation flow

* fixed pr review suggested issues

* removed service project layer

* fixed issues

* delete unnessary files

* added requested changes

* fix issue in prompt config toggle

* standalone agentic loop module

* fixed requested changes

* fixed ruff format issues

* complete API semantic searcher with ambiguous result handling and tool classifier routing

* complete semantic searcher evaluation and update to multi point indexing strategy

---------

Co-authored-by: Thiru Dinesh <56014038+Thirunayan22@users.noreply.github.com>

Author: nuwangeek

DSL/Ruuter.public/rag-search/POST/api-tools/search.yml

@@ -30,10 +30,19 @@ class ApiToolIndexerConstants:
     RETRY_DELAY_BASE = 2  # Exponential backoff base (2^attempt seconds)
     REQUEST_TIMEOUT = 60  # seconds
 
+    # Number of example queries generated per endpoint.
+    # Each example becomes its own Qdrant point so its vector sits in the exact
+    # language region of the embedding space, enabling short-query matching.
+    EXAMPLE_QUERY_COUNT = 5
+
     # Context Enrichment Template
-    # Mirrors the service workflow (intent_data_enrichment/constants.py).
     # Full template goes in chunk_prompt; document_prompt is left empty.
-    # The LLM summarises the chunk content into a rich semantic context.
+    #
+    # Multi-point indexing strategy:
+    #   - Each example query line is extracted and stored as its own Qdrant point,
+    #     embedded from that individual sentence alone.
+    #   - The prose + all examples combined become one summary point.
+    # All in the same language as the endpoint description — no bilingual duplication.
     CONTEXT_TEMPLATE = """<document>
 {full_endpoint_info}
 </document>
@@ -52,18 +61,17 @@ class ApiToolIndexerConstants:
 - Related concepts and use cases
 - Common ways users might ask for this functionality in natural language
 
-Then, on a new line, add a section exactly as shown below with 6 to 8 realistic and diverse example questions a real user might ask when they need this endpoint. Cover different phrasings, synonyms, and indirect ways of asking — do not just repeat the description verbatim.
+IMPORTANT: Generate the prose context and the example questions in the SAME LANGUAGE as the endpoint description above. However, always use the exact section header "Example queries:" in English regardless of language — this is a required machine-readable marker.
 
 IMPORTANT for example queries: This is a system built for Estonian government digital services (Bürokratt). Ground the examples in an Estonian context — use Estonian cities (Tallinn, Tartu, Pärnu, Narva), Estonian institutions, and Estonia-relevant scenarios. Only use non-Estonian locations if the endpoint is explicitly about comparing or fetching data for multiple countries.
 
+Then add a section with exactly {example_count} realistic and diverse example questions a real user might ask when they need this endpoint. Cover different phrasings, synonyms, and indirect ways of asking — do not just repeat the description verbatim.
+
 Example queries:
 - <example question 1>
 - <example question 2>
 - <example question 3>
 - <example question 4>
 - <example question 5>
-- <example question 6>
-
-IMPORTANT: Generate everything in the SAME LANGUAGE as the endpoint description above. If the description is in Estonian, respond in Estonian. If in English, respond in English. If in Russian, respond in Russian.
 
 Answer only with the enriched context and example queries — nothing else."""