Speech-to-Insight

I built this because I kept finding myself with long recordings — interviews, meetings, podcasts — and no good way to quickly understand what was actually in them without listening to the whole thing. So this pipeline takes an audio file and pulls out the stuff that matters: who was mentioned, what topics came up, which sentences best represent the whole thing, and a word cloud for a quick visual scan.

It's entirely local. No APIs, no cloud, no sending your audio anywhere. Everything runs on your machine.

---

What it does

Drop in an audio file, and the pipeline runs five things in sequence:

1. Transcription — Whisper converts speech to text with word-level timestamps
2. Speaker diarization — pyannote.audio figures out who was speaking and when
3. Named entity recognition — spaCy identifies people, organisations, places, dates, and money figures
4. Topic modeling — BERTopic discovers the main themes running through the audio
5. Summarization — LexRank picks the sentences that best represent the overall content

The results get merged into a single JSON file and you can either query them through a REST API or explore them in a Streamlit dashboard.

---

The concepts, explained

Whisper (transcription)

Whisper is OpenAI's speech recognition model, released as open source. It's trained on a huge amount of multilingual audio scraped from the web, which makes it surprisingly robust — it handles accents, background noise, and technical vocabulary better than most alternatives.

The model comes in several sizes: tiny, base, small, medium, large. Bigger models are more accurate but take longer and need more RAM. This project defaults to base, which is about 150MB and a decent middle ground. You can change the model in config.yaml.

One thing worth knowing: Whisper doesn't just give you a wall of text. It gives you segments (roughly sentence-length chunks with start/end timestamps) and, if you ask for it, word-level timestamps. We use both — segments feed into the topic model, word timestamps let us place named entities on a timeline.

spaCy NER (named entity recognition)

Named entity recognition is the task of identifying and classifying "named things" in text — people, companies, locations, dates, monetary amounts, etc. spaCy's NER is a neural model trained to label spans of text with entity types.

The way it works under the hood: the model reads the text as a sequence of tokens, runs them through a transformer-style encoder, and learns to predict whether each token starts, continues, or ends a named entity, and what type it is. This is called BIO tagging (Beginning, Inside, Outside).

We run spaCy on the full transcript text and filter for entity types defined in config.yaml (PERSON, ORG, GPE, DATE, MONEY, PRODUCT). We also do a rough timestamp lookup: for each entity, we find the first word in the Whisper word-timestamp list and use that as the time it appeared. It's approximate but good enough for the timeline chart.

You need to download the English model once before running:

python -m spacy download en_core_web_sm

pyannote.audio (speaker diarization)

Diarization answers the question "who spoke when?" It's a harder problem than it sounds. You have to detect speaker changes in a continuous audio stream, extract a voice embedding for each detected segment, cluster those embeddings into groups (one per speaker), and output a timeline of who was active at each moment.

pyannote.audio is the current state-of-the-art open-source library for this. The pipeline it uses internally does three things:

1. Voice activity detection — finds the parts of the audio that contain speech, as opposed to silence, music, or noise
2. Speaker segmentation — within the speech regions, detects speaker change points
3. Speaker embedding + clustering — extracts a compact voice fingerprint for each segment, then clusters them so segments from the same person end up in the same cluster

The model is pyannote/speaker-diarization-3.1, which lives on HuggingFace. It's free but you have to create an account, accept the model's terms of use, and generate a read token. The token goes in config.yaml or in a HF_TOKEN environment variable (don't commit a real token to your repo).

Once we have the raw diarization output — a list of {speaker, start, end} segments — we align it with the Whisper transcript. For each Whisper segment, we find whichever speaker had the most overlap during that time window and assign it. Then we merge consecutive same-speaker segments into conversation turns, giving you a clean labelled dialogue.

To enable:

# config.yaml
diarization:
  enabled: true
  hf_token: null  # leave null and export HF_TOKEN=hf_... instead

export HF_TOKEN=hf_your_token_here

The first run downloads the model (~1GB) and caches it. Subsequent runs are much faster. GPU is used automatically if available, which makes a big difference on long recordings.

BERTopic (topic modeling)

Topic modeling is the problem of discovering what a document (or set of documents) is about without being told in advance. Classic approaches like LDA treat documents as bags of words and look for co-occurring word patterns. BERTopic does something smarter.

The BERTopic pipeline has four steps:

1. Embed — each sentence gets turned into a vector using a sentence-transformer model (by default all-MiniLM-L6-v2). These vectors capture semantic meaning, so "car" and "vehicle" end up close together in the embedding space.

2. Reduce dimensions — embeddings are 384-dimensional, which is too high for clustering to work well. UMAP reduces them to 5 dimensions while preserving local structure.

3. Cluster — HDBSCAN groups the reduced vectors into clusters. Sentences that talk about similar things end up in the same cluster. Outliers get assigned to topic -1.

4. Extract keywords — for each cluster, BERTopic uses c-TF-IDF (class-based TF-IDF) to find the words that are distinctive for that topic compared to all other topics.

The result is a set of topics, each described by keywords and containing a list of sentences. We split the Whisper transcript into sentences using the segment boundaries (each Whisper segment is roughly one sentence), then run BERTopic on those.

One thing to note: BERTopic needs a minimum number of sentences to work well. If your audio is very short (fewer than 5 segments), topic modeling is skipped.

LexRank (summarization)

LexRank is an extractive summarization algorithm. Unlike abstractive methods (which generate new sentences), LexRank picks actual sentences from the source text. There's no LLM involved — it's graph-based.

The idea comes from PageRank. Here's how it works:

• Build a graph where each sentence is a node
• Draw edges between every pair of sentences, weighted by their cosine similarity (how similar their TF-IDF vectors are)
• Run the PageRank algorithm on this graph
• Sentences that are similar to many other sentences score highly — they're central to the content

The intuition is that sentences which say something many other sentences also say (but more concisely) are likely to be the most representative. It works well for transcripts because spoken language tends to be repetitive.

We use the sumy library for this. The number of sentences returned is configured in config.yaml under nlp.summary_sentences (default 5).

FastAPI (the API)

FastAPI is a Python web framework built on top of Starlette and Pydantic. It's fast (async by default), generates OpenAPI docs automatically, and handles request validation cleanly.

The API has four endpoints:

• POST /analyze — upload an audio file, get back the full insight JSON
• GET /insights/{stem} — retrieve previously processed results by filename
• GET /insights — list all processed files
• GET /health — liveness check

When you upload a file, it saves it to data/raw/, runs the full pipeline, and streams the result back. The Swagger UI at /docs is the easiest way to try it interactively.

Streamlit (the dashboard)

Streamlit is a Python library that turns scripts into web apps. You write normal Python, add st. calls where you want UI elements, and it handles the rendering. It's not great for production apps with complex state, but it's excellent for data exploration dashboards like this one.

The dashboard reads the processed insight JSON files from data/processed/insights/, lets you pick one from a sidebar dropdown, and visualises:

• Key metrics (duration, word count, entity count, topic count)
• The LexRank key sentences
• A word cloud generated from word frequencies
• Named entity breakdown by type, with a bar chart of the top 15 entities by mention count
• A topic breakdown showing how many sentences belong to each topic
• An entity timeline scatter plot showing when entities appear in the audio
• Speakers section — per-speaker stats (total speaking time, word count, turn count) and a Gantt-style chart showing who was talking when
• Conversation view — the full diarized transcript as a scrollable dialogue, color-coded by speaker with timestamps
• The full transcript in an expander at the bottom

---

Setup

You need Python 3.11 or later and ffmpeg installed on your system (Whisper uses it to decode audio).

Install ffmpeg:

• Mac: brew install ffmpeg
• Ubuntu/Debian: sudo apt install ffmpeg
• Windows: download from ffmpeg.org and add to PATH

Install Python dependencies:

pip install -r requirements.txt

Download the spaCy language model:

python -m spacy download en_core_web_sm

The first time you run the pipeline, Whisper will also download the model weights (~150MB for base). They get cached in ~/.cache/whisper so subsequent runs are fast.

---

Usage

Process an audio file:

python run_pipeline.py --audio data/raw/your_file.mp3

Process and immediately start the dashboard + API:

python run_pipeline.py --audio data/raw/your_file.mp3 --serve

Start the services without processing (if you've already run the pipeline):

python run_pipeline.py --serve-only

Start them individually:

# API (Swagger UI at http://localhost:8000/docs)
uvicorn src.api.main:app --reload

# Dashboard
streamlit run dashboard/app.py

---

Output

Processing creates intermediate files and one final file per audio:

data/processed/transcripts/your_file_transcript.json    # Whisper output
data/processed/insights/your_file_diarization.json      # pyannote speaker turns
data/processed/insights/your_file_entities.json         # spaCy NER results
data/processed/insights/your_file_topics.json           # BERTopic results
data/processed/insights/your_file_summary.json          # LexRank + word frequency
data/processed/insights/your_file_insights.json         # merged, this is what the API/dashboard reads

The final insights JSON structure:

{
  "audio_file": "interview.mp3",
  "language": "en",
  "duration_seconds": 1842,
  "total_words": 4201,
  "transcript": "...",
  "key_sentences": ["...", "..."],
  "word_frequency": {"machine": 47, "learning": 31},
  "entities": [{"text": "OpenAI", "label": "ORG", "count": 12, "first_seen_seconds": 34.2}],
  "entities_by_type": {"ORG": ["OpenAI"], "PERSON": ["..."]},
  "entity_timeline": [{"entity": "OpenAI", "label": "ORG", "time_seconds": 34.2}],
  "topics": [{"id": 0, "label": "Topic 0: model, training, data", "keywords": ["..."], "sentence_count": 23}],
  "sentence_topics": ["..."],
  "total_topics": 4
}

---

Configuration

Everything is controlled through config.yaml. The main things you might want to change:

transcription:
  model: "base"        # tiny / base / small / medium / large
  language: null       # null = auto-detect, or force e.g. "en", "de", "fr"

nlp:
  summary_sentences: 5   # how many key sentences to extract
  topic_model:
    nr_topics: "auto"    # or an integer if you want a fixed number of topics
    min_topic_size: 3    # minimum sentences per topic

Larger Whisper models (small, medium, large) are significantly more accurate for technical content, accented speech, or non-English audio. The tradeoff is speed and memory — large needs around 10GB of RAM and takes several minutes per hour of audio on CPU.

diarization:
  enabled: true          # flip to true to activate
  hf_token: null         # set here or as HF_TOKEN env var
  min_speakers: null     # if you know how many speakers, set it — improves accuracy
  max_speakers: null     # same — bounding this helps the clustering step

If you know the number of speakers in advance, setting both min_speakers and max_speakers to the same value locks the model in and consistently gives better results than auto-detection.

---

Project structure

├── src/
│   ├── transcribe/      # Whisper integration
│   ├── diarize/         # pyannote speaker diarization
│   ├── nlp/             # entities, topics, summarize — one module each
│   ├── pipeline/        # orchestrates all five steps
│   └── api/             # FastAPI app
├── dashboard/           # Streamlit dashboard
├── tests/               # unit tests for NLP modules
├── data/
│   ├── raw/             # put your audio files here
│   └── processed/       # pipeline outputs land here
├── config.yaml          # single source of truth for all settings
└── run_pipeline.py      # CLI entry point

Each NLP module (entities.py, topics.py, summarize.py) exposes a run(transcript, config_path) function. pipeline/run.py calls them in order and merges the results. If you want to swap out any component — say, replace LexRank with something else — you just need to make that module's run() return the same dict shape.

---

Running the tests

pytest tests/

The tests cover the NLP utility functions (sentence extraction, word frequency, segment filtering) and don't require any models or audio files to be present. They run fast.