A prototype speech recognition model that connects a speech encoder to a text-only language model. A frozen Whisper encoder turns audio into features, a small trainable connector maps them into the embedding space of a frozen LLM, and the LLM generates the transcript.
waveform (16 kHz, padded or cut to 30 s)
-> WhisperFeatureExtractor (log-mel spectrogram)
-> Whisper encoder, openai/whisper-small (frozen)
-> AudioConnector (trainable): Conv1d, 4x shorter in time -> MLP -> LayerNorm
-> Qwen/Qwen2.5-7B (frozen, loaded in 8-bit), fed as inputs_embeds
-> transcript (greedy decoding)
- Data.
get_dataset.pydownloads the English (en_us) subset of FLEURS (google/fleurs) from Hugging Face, resamples it to 16 kHz, and writesdata/fleurs/{train,validation,test}.csv. - Audio.
AudioProcessorloads mono audio with librosa at 16 kHz, returnsNonefor clips shorter than 0.1 s, pads or truncates to 30 s, and peak-normalizes. - Dataset.
AudioDatasetreads a CSV and returns each waveform with its transcript, tokenized by the tokenizer passed in (padded or truncated to 512 tokens). - Connector.
AudioConnectorcompresses the encoder output in time with a stridedConv1d(kernel and stride equal tocompression_factor, default 4), then projects it withLinear -> GELU -> Dropout -> Linear -> LayerNormto the LLM hidden size.MultiModalASRsets the input and output sizes from the two model configs. - Model.
MultiModalASRloads both pretrained models and freezes them by default, so only the connector is trained. It also enables gradient checkpointing on the LLM.generate()decodes greedily and returns a list of strings.
| Path | Contents |
|---|---|
| data_module/get_dataset.py | Download FLEURS en_us and write one CSV per split |
| data_module/audio_processor.py | AudioProcessor: load, resample, pad or trim, normalize |
| data_module/dataset.py | AudioDataset (CSV to waveform plus tokens) and collate_fn |
| models/audio_connector.py | AudioConnector: encoder features to LLM embeddings |
| models/multimodel_asr.py | MultiModalASR: encoder, connector, and LLM with forward and generate |
| pyproject.toml, uv.lock | Dependencies, managed with uv (Python 3.10) |
git clone https://github.com/karthikbharadhwajKB/Audio-LLMs.git
cd Audio-LLMs
uv syncOn platforms other than Linux, pyproject.toml installs torch from the PyTorch CUDA 12.6 wheel index. Loading the LLM in 8-bit (the default) also needs bitsandbytes and accelerate, which are not listed in pyproject.toml:
uv pip install bitsandbytes accelerateThere is no training or evaluation script yet. Run commands from the repository root, since the modules import each other as data_module.* and models.*.
Download FLEURS English and write the CSVs to data/fleurs/ (the data/ folder is git-ignored):
uv run python data_module/get_dataset.pyThe audio processor and the connector can be used on their own:
import torch
from data_module.audio_processor import AudioProcessor
from models.audio_connector import AudioConnector
processor = AudioProcessor(sample_rate=16_000, max_duration=30.0)
waveform = processor.preprocess_single("clip.wav") # tensor of 480000 samples, or None
connector = AudioConnector(whisper_dim=768, llm_dim=4096, compression_factor=4)
features = torch.randn(1, 1500, 768) # Whisper-small encoder output for 30 s
embeddings = connector(features) # shape (1, 375, 4096)Early prototype, August 2025. The data pipeline and model classes are written, but the full model has not been trained or run end to end. Known gaps in the current code:
MultiModalASR.__init__callsself.log_model_info(), but the method is named_log_model_info, so building the model fails after both pretrained models have loaded.forward()passes only the audio embeddings to the LLM asinputs_embeds, together with the textlabels. The transcript embeddings are not appended yet, so the lengths do not match (375 audio positions against 512 label tokens with the defaults) and the loss computation raises an error.get_dataset.pynames the audio columnaudio_file, whileAudioDatasetexpectsaudio_pathby default (passaudio_column="audio_file").collate_fnreadsitem["transcription"], but dataset items store the text undertexts.
Apache License 2.0. See LICENSE.