TechMart Enterprise Voice Agent

Blog

https://platformatory.io/blog/techmart-telephony-ai/

Overview

The TechMart Enterprise Voice Agent is a real-time, highly scalable AI customer support agent built to handle telecom interactions. It bridges Vobiz.ai WebSocket audio streams with a stateful LangGraph core, utilizing Pipecat for low-latency voice streaming and Sarvam AI for native Indian dialect Speech-to-Text (STT) and Text-to-Speech (TTS).

It features robust dynamic tool calling, semantic hybrid search against ClickHouse vector database (using local all-MiniLM-L6-v2 embeddings), and real-time observability via Langfuse.

---

Technology Stack

Component	Technology	Description
Framework	FastAPI	Hosts the Admin CMS, WebSockets, and webhook endpoints.
Voice Transport	Pipecat-AI	Low-latency audio pipeline managing WebSocket and Vobiz streams.
Cognitive Engine	LangGraph & LangChain	Manages multi-turn state, LLM routing, and tool execution.
Primary LLM	Groq (llama-3.3-70b-versatile)	Fast, high-reasoning LLM for the main agent brain.
Summary LLM	Groq (llama-3.1-8b-instant)	Background task LLM for summarizing conversation memory.
Speech Services	Sarvam AI	saaras:v3 (STT) and bulbul:v3 (TTS) for Indian dialects.
Vector Database	ClickHouse	Fast, real-time analytics and vector hybrid search.
Local Embeddings	Sentence-Transformers	CPU-bound all-MiniLM-L6-v2 for generating search vectors.
Testing	Pytest	Automated API and Graph logic testing.

---

Local Development Setup

Follow these instructions to configure and set up the TechMart Voice Agent locally.

1. Clone the Repository

git clone https://github.com/Platformatory/techmart_telephony_ai.git
cd techmart_telephony_ai

2. Environment Setup

Initialize and activate your virtual environment:

python -m venv venv
.\venv\Scripts\activate

Install the required dependencies:

pip install -r requirements.txt

3. Environment Variables

Create a .env file in the root directory and populate it with your API keys:

# Database (ClickHouse)
CLICKHOUSE_HOST=your_clickhouse_host
CLICKHOUSE_PORT=your_clickhouse_port
CLICKHOUSE_USER=your_username
CLICKHOUSE_PASSWORD=your_password

# Telemetry (Langfuse)
LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
LANGFUSE_HOST=your_langfuse_host

# LLM & Speech APIs
GROQ_API_KEY=your_groq_key
SARVAM_API_KEY=your_sarvam_key

# Telecom API (Vobiz)
VOBIZ_AUTH_ID=your_vobiz_id
VOBIZ_AUTH_TOKEN=your_vobiz_token
PUBLIC_URL=https://your-ngrok-url.ngrok-free.app

4. Database Setup (ClickHouse)

The agent relies on ClickHouse. The tables will be automatically verified via src/db/schema.py, but ensure your cluster is alive. The system utilizes:

• product_catalog (Hybrid Search for Products)
• company_faqs (Knowledge Base)
• company_tos (Terms of Service)
• customers & order_history (CRM)
• call_tickets & complaint_tickets (Call Logs)

You can manage this data directly via the built-in Admin CMS once the server starts.

---

Running the Application

1. Live Server (Vobiz.ai Phone Integration)

The primary way this application runs is by exposing a FastAPI server that acts as a webhook and WebSocket host for incoming Vobiz telecom phone calls.

First, start the FastAPI application:

python -m src.main

Because Vobiz needs to reach your local server from the public internet to bridge the phone call audio, you must use a tunneling service like Ngrok:

# In a separate terminal
./ngrok http 8000

Take the Ngrok URL (e.g., https://1234-abcd.ngrok-free.app) and configure your Vobiz account to point its webhooks to https://<your-ngrok-url>/vobiz-xml.

Once configured, any live phone call to your Vobiz number will instantly stream 8kHz audio into your Pipecat pipeline!

• Admin CMS Dashboard: http://localhost:8000/admin (Password: dev-secret-key)

2. Text-Mode Tester (No Audio / Phone Required)

If you want to rapidly test the AI's internal logic, tool execution, and database connectivity without setting up Vobiz or making a real phone call, you can use the interactive text tester:

python test_app.py

This isolates the LangGraph core and allows you to type directly to the agent.

3. Automated Testing Suite

To run the Pytest suite (tests LangGraph core logic and FastAPI endpoints):

python -m pytest tests/ -v

---

Architecture & Pipeline Deep Dive

The TechMart Voice Agent is not just an LLM script; it is a complex, multi-threaded pipeline designed for zero-latency voice interactions over phone networks.

1. The Audio Pipeline (Pipecat)

When a call arrives via Vobiz, src/main.py provisions a WebSocket. src/bot/pipeline.py constructs a PipelineWorker that strings together:

• Transport: FastAPIWebsocketTransport (Receives 8kHz audio).
• VAD & Barge-In: Local SileroVADAnalyzer handles turn-taking. If the user interrupts the AI (barge-in), the Pipecat transport immediately stops the TTS stream and cancels the LangGraph execution task to listen to the user.
• Back-Channeling Tolerance: The pipeline utilizes MinWordsUserTurnStartStrategy(min_words=2). This means one-word interjections (like "yes", "okay") are intentionally ignored and will not trigger a barge-in, allowing the AI to naturally finish its sentence.
• STT: SarvamSTTService transcribes the audio.
• Context Aggregator: LLMContextAggregatorPair bundles the transcribed text and passes it to the LangGraph Adapter.
• TTS: SarvamTTSService converts the AI's response back into 8kHz audio.

2. The LangGraph Adapter (src/bot/adapter.py)

Pipecat natively supports OpenAI, but we use a custom stateful LangGraph engine. The LangGraphLLMService subclasses Pipecat's LLM interface. When VAD triggers, the adapter:

1. Gathers the conversation history.
2. Starts the TTFB (Time-To-First-Byte) tracker.
3. Fires a "filler word" (e.g., "Just a moment...") asynchronously — but only if the LLM takes longer than 1.5 seconds to respond. For fast answers, the real response starts immediately with no filler.
4. Streams the AI response chunks from LangGraph directly into the TTS engine.

3. The LangGraph State Machine (The "Brain")

The core cognitive engine is built on LangGraph (src/graph/workflow.py). Rather than passing stateless text strings, we pass a highly structured AgentState dictionary (src/graph/state.py).

A. Context Injection & System Prompts Every time the LLM evaluates the state, the agent_node dynamically injects real-time context into the system prompt:

• Customer CRM Data: It injects the user's name and customer_id.
• Dynamic Empathy: It reads the user_emotion flag from the audio Sentiment Processor. If the caller's detected emotion is anything other than neutral, the prompt instructs the 70B LLM to adopt a more empathetic and reassuring tone.
• Native Multilingualism: It reads the detected_language. Instead of piping text through a slow translation API, the prompt instructs the Groq Llama-3 70B model to reply natively in the detected language (e.g., Hindi or Tamil) while executing English database queries in the background.

Supported Languages (detected automatically from the caller's speech via Sarvam STT):

Language Code	Language
en-IN	English
hi-IN	Hindi
kn-IN	Kannada
ml-IN	Malayalam
ta-IN	Tamil
te-IN	Telugu
bn-IN	Bengali
gu-IN	Gujarati
mr-IN	Marathi
pa-IN	Punjabi
od-IN	Odia

B. Tool Calling & Security (InjectedState) The brain is equipped with 8 ClickHouse database tools (e.g., fetching orders, logging complaints). To prevent the LLM from hallucinating database queries for the wrong user, we use LangGraph's InjectedState. The LLM is only allowed to ask for the tool; the customer_id is securely injected into the function call at the Python level from the secure CRM state, guaranteeing the user can only access their own data.

C. Rolling Memory Summarization To prevent context window bloat (which slows down TTS latency), the graph utilizes a dual-LLM architecture. When the conversation exceeds 6 turns, the should_continue router redirects to a summarize_conversation_node before ending the turn. This node uses a smaller, ultra-fast model (Llama-3.1-8b-instant) to compress older messages into a bulleted summary, which is stored in the graph state for future turns.

D. MemorySaver The graph uses LangGraph's native MemorySaver checkpointer, tying the session_id to the thread so that tool results and context persist perfectly across the entire phone call without amnesia.

E. State Machine Flowchart

graph TD
    %% Base Graph Nodes
    S((START))
    Agent[agent_node <br> Llama-3.3-70b]
    Tools[tool_node <br> ClickHouse Executions]
    Summary[summarize_conversation_node <br> Llama-3.1-8b]
    E((END))

    %% Graph Edges (workflow.py)
    S --> Agent
    
    Agent -- "should_continue: 'tools'" --> Tools
    Tools -- "returns tool_message" --> Agent
    
    Agent -- "should_continue: 'summarize' (len > 6)" --> Summary
    Summary --> E
    
    Agent -- "should_continue: END" --> E

    %% Tool Bindings Visualization
    subgraph Bound Database Tools
        Tools --- T1[Hybrid RRF Search: search_catalog, get_support_faq]
        Tools --- T2[Vector Search: search_legal_tos]
        Tools --- T3[SQL Query: get_order_status, get_customer_history]
        Tools --- T4[SQL Write/Logic: check_complaint_eligibility, raise_ticket]
        Tools --- T5[State Flag: escalate_to_human]
    end

Loading

4. Vector Search & ClickHouse Concurrency

The application performs hybrid RAG searches using cosine distance. Because ClickHouse HTTP clients are not natively async, all database queries in src/graph/tools.py are wrapped in asyncio.to_thread and use threading.local() connection pooling to prevent the Python GIL from blocking the real-time audio event loop.

5. Post-Call Processing

When the user hangs up (or the agent transfers the call), a background task fires to summarize the entire transcript and write a call_ticket back to ClickHouse, complete with vector embeddings for future analytics.

6. Human Phone Handoff (Vobiz XML)

If the caller is extremely frustrated or explicitly requests to speak to a human, the LangGraph brain executes the escalate_to_human tool. When this happens, the Pipecat adapter (not FastAPI directly) makes an outbound POST call to the Vobiz mid-call REST API, instructing Vobiz to redirect the active call leg to the /transfer-to-human endpoint on our server. The adapter then immediately plays a ringback tone to the caller and pushes an EndFrame to close the AI WebSocket. When Vobiz calls back on POST /transfer-to-human, the FastAPI server responds with XML containing a <Speak> hold message followed by a <Dial> tag that bridges the caller directly to the human agent's real phone number (TRANSFER_AGENT_NUMBER).

7. Full Sequence Diagram

sequenceDiagram
    participant User
    participant Vobiz as Vobiz Telecom
    participant Main as FastAPI (main.py)
    participant Pipe as Pipecat (pipeline.py)
    participant STT as Sarvam STT
    participant LangGraph as LangGraph Core
    participant DB as ClickHouse DB
    participant TTS as Sarvam TTS
    participant Human as Human Agent

    User->>Vobiz: Dials Phone Number
    Vobiz->>Main: POST /vobiz-xml
    Main-->>Vobiz: Returns XML with wss:// URI
    Vobiz->>Main: Connects WebSocket (/ws/vobiz)
    Main->>DB: Fetch Customer Profile (asyncio.to_thread)
    DB-->>Main: Returns Profile
    Main->>Pipe: Initializes WorkerRunner & Pipeline
    
    Note over Pipe,TTS: The Greeting
    Pipe->>TTS: "Hello [Name], welcome to TechMart."
    TTS-->>Vobiz: 8kHz Audio Stream
    Vobiz-->>User: Hears Greeting
    
    Note over User,TTS: Active Conversation Loop
    User->>Vobiz: Speaks ("My laptop broke!")
    Vobiz->>Pipe: Streams raw audio bytes
    Pipe->>STT: Audio bytes
    STT-->>Pipe: Transcribed Text + Language Detection
    Pipe->>LangGraph: User stopped speaking (VAD trigger)
    LangGraph-->>TTS: Filler word ("Just a moment...")
    TTS-->>Vobiz: Audio
    
    LangGraph->>DB: Tool Execution (e.g., get_customer_history)
    DB-->>LangGraph: Tool Results (Order Data)
    LangGraph-->>TTS: Streams AI Response Tokens
    TTS-->>Vobiz: Synthesized 8kHz Audio Stream
    Vobiz-->>User: Hears Response
    
    alt Call Completes Normally
        User->>Vobiz: Hangs Up
        Vobiz->>Main: WebSocket Disconnects
    else AI Escalates to Human
        LangGraph->>Pipe: Sets handoff_status = Accepted
        Pipe->>Vobiz: POST Vobiz mid-call API (redirect aleg to /transfer-to-human)
        Pipe->>Vobiz: Plays ringback tone audio + closes AI WebSocket (EndFrame)
        Vobiz->>Main: POST /transfer-to-human
        Main-->>Vobiz: Returns XML with <Speak> hold message + <Dial> Agent Number
        Vobiz->>Human: Bridges Call to Human Agent
        Human-->>User: "Hi, I'm taking over..."
    end
    
    Main->>LangGraph: Extract memory & summarize
    LangGraph->>DB: Writes Call Ticket & Vector Embedding

Loading