WhisperX is one of those repositories that looks small from the outside — a Python package and a CLI called whisperx — but underneath it coordinates three very different machine learning models against the same audio buffer. A faster-whisper CTranslate2 engine does the speech recognition, a wav2vec2 phoneme model anchors every word to an exact time range, and a pyannote speaker model answers the “who spoke when” question. The reason people reach for it is simple: vanilla Whisper decodes audio sequentially and its timestamps are only segment-accurate, while WhisperX batches the decoding and then recovers word-level timestamps with a second, cheaper model.

Reading the source matters because the interesting engineering is in the handoffs. Voice activity detection decides where speech actually is, those regions get packed into up-to-30-second chunks, the chunks fly through the recognizer in parallel batches, and only then does the aligner see the transcript. If you have ever wondered why your transcription tool hallucinates on silence or drops words at chunk boundaries, the answers are in this code. The design also explains WhisperX’s memory discipline: each stage explicitly unloads its model and calls gc.collect() before the next one loads, so a laptop GPU can survive the whole sequence.

This post walks the repository the way we have been touring other inference projects — Piper for on-device TTS, ONNX Runtime for portable model execution — except here the subject is speech recognition, and the tour ends at subtitle files rather than audio. Two diagrams were generated directly from the repository tree: a compact overview and a detailed call-level map. Every symbol mentioned below links to the real line in the upstream source on main.

Architecture at a Glance

From CLI entry through voice activity, batched recognition, alignment and speaker labeling to the output writers.

Reading the overview from left to right:

• The journey starts at cli, L12, which builds the entire argparse surface — model size, device, compute type, VAD thresholds, diarization flags — and dispatches into the task function.
• load_audio, L25 decodes any container format through ffmpeg into 16 kHz mono float32, and the module-level SAMPLE_RATE, L13 constant is the contract every later stage relies on.
• load_model, L462 is the factory that wires the faster-whisper engine, tokenizer defaults and one of the two VAD backends into a single object.
• The Vad, L7 base class defines the backend contract, and its static merge_chunks, L20 helper packs detected speech regions into batchable chunks.
• Two backends implement that contract: load_vad_model, L21 for the default pyannote route and Silero, L18 for the lightweight alternative.
• transcribe, L257 is where batching happens: chunks become a DataLoader, and each batch is encoded and decoded in parallel on the GPU.
• align, L152 then runs wav2vec2 forced alignment over the same waveform to pin words to milliseconds.
• When --diarize is passed, assign_word_speakers, L185 overlaps those words with pyannote speaker turns.
• Finally get_writer, L443 selects the output format — srt, vtt, tsv, json, txt or Audacity labels.

From Install to Your First Model

Installation is one line, plus ffmpeg on your PATH:

pip install whisperx

Then the CLI does the whole sequence in one shot:

whisperx audio.mp3 --model large-v2 --batch_size 16 \
  --compute_type float16 --diarize --hf_token YOUR_HF_TOKEN \
  --output_format all

Three flags deserve attention. --batch_size controls how many VAD chunks are decoded in parallel — this is the main speed lever, and raising it is safe precisely because chunks are independent. --vad_method switches between pyannote (default) and silero; the code in load_model treats a manually supplied VAD model as higher priority than the method flag. --diarize pulls in a gated pyannote model, which is why an HF token is needed.

The same sequence works as a library, which is what most production code should use:

import whisperx, gc

device = "cuda"
model = whisperx.load_model("large-v2", device, compute_type="float16")
audio = whisperx.load_audio("audio.mp3")
result = model.transcribe(audio, batch_size=16)
del model; gc.collect()          # free Whisper before loading the aligner

align_model, meta = whisperx.load_align_model(result["language"], device)
result = whisperx.align(result["segments"], align_model, meta, audio, device)

The Python surface mirrors the CLI’s three stages exactly, and that mirrors the internal call graph: load_model returns the engine wrapper, transcribe returns a plain dict matching the TranscriptionResult, L65 TypedDict, and align upgrades it in place to the aligned shape. Note the explicit del and gc.collect() between stages — when you orchestrate the models yourself, the memory handoff is your responsibility, not the library’s. The package/__init__.py exports only the handful of names above, which is a quiet design statement: the intended composition is short, and anything fancier should go through the CLI.

What comes back depends on the format. The JSON dump mirrors the AlignedTranscriptionResult, L73 TypedDict — segments, and inside each one a words array where every entry carries start, end, word and, after diarization, speaker. The srt and vtt files are rendered from the same structure by the subtitle writer, which is why word-level highlighting works there too.

Under the hood the engine class extends faster_whisper.WhisperModel directly (see WhisperModel, L32), so anything CTranslate2 supports — float16 on CUDA, float32 on CPU, int8 quantization via --compute_type — carries over unchanged.

The Batched Inference Loop

The detailed map: VAD chunking feeds a DataLoader-style iterator, whose batches flow through encode and batched beam-search decode, then onward to alignment and diarization.

The heart of the repo is what happens inside transcribe, L257. First the VAD backend runs over the waveform; its onset and offset thresholds (defaults 0.500 and 0.363, tunable via --vad_onset/--vad_offset) decide what counts as speech. The detected regions are then merged by merge_chunks into windows of at most chunk_size seconds — 30 by default. Crucially, the code checks the backend with issubclass(type(self.vad_model), Vad) and falls back to the pyannote preprocessing path otherwise, which is what lets you plug in a custom VAD.

Those chunks feed get_iterator, L236, a torch DataLoader over slice indices computed with SAMPLE_RATE arithmetic, so workers pull raw audio slices lazily instead of copying the whole buffer. Each batch goes through encode, L121, which pads log-mel features into a CTranslate2 StorageView, and lands in generate_segment_batched, L38 for batched beam-search decoding. If you never passed --language, detect_language, L447 runs once per file as a fallback and the tokenizer is built from the answer.

One subtlety worth stealing: when --suppress_numerals is set, the decoder suppresses numeric tokens (the helper lives at the top of asr.py). Why? Because wav2vec2 phoneme models are bad at aligning digits — “2026” spoken aloud has no written form to match — so WhisperX prefers spelled-out text it can align faithfully, and later stages fix the rendering.

The stage boundaries are also where memory gets reclaimed. After the last file is transcribed, the code deletes the ASR object, calls gc.collect() and empties the CUDA cache before loading the alignment model. Same after alignment, before diarization. It reads like ceremony, but on a 6 GB GPU it is the difference between finishing and OOM.

wav2vec2 Forced Alignment, Then Speaker Labels

Alignment is where WhisperX earns its name. The DEFAULT_ALIGN_MODELS_HF, L71 registry maps each language to a wav2vec2-style phoneme checkpoint (English gets a wav2vec2 model; Japanese and Chinese, which lack spaces, live in a separate constant and use different word-joining rules). load_align_model, L111 fetches the right one — and if the transcript’s language differs from the aligner’s, the CLI reloads a matching aligner mid-run.

The algorithm inside align, L152 is a textbook Viterbi forced alignment: run the wav2vec2 model over the audio to get per-frame phoneme probabilities, build a frame-by-token trellis with get_trellis, L548, recover the best path with backtrack, L574, collapse repeated tokens into spans with merge_repeats, L625, and join spans into words with merge_words, L643. Words the aligner could not place get NaN timestamps, which interpolate_nans, L470 fills by nearest or linear interpolation depending on --interpolate_method.

Diarization is deliberately thin. The wrapper in diarize.py loads a pyannote speaker model (community release 1 is the current default name in the CLI), runs it over the file, and returns turn annotations as Segment, L266 objects — start, end, speaker. assign_word_speakers, L185 then indexes those turns and walks every aligned word through an IntervalTree, L14 to find which turn overlaps it most; the winning speaker label is written onto the word and majority-voted upward to the parent segment. No embeddings dance unless you ask for --speaker_embeddings.

Honest Limits

The alignment chain inherits wav2vec2’s weaknesses. Digits, acronyms and code-switched speech align poorly — hence the numeral suppression trick — and exotic languages fall back to multilingual checkpoints with noticeably sloppier boundaries. If no per-language aligner exists, --no_align degrades you back to segment-level timestamps.

VAD is a hard gate, not a suggestion. Music-heavy or overlapping-speech audio produces merged or missing chunks, and everything downstream inherits that blindness. The chunking heuristic (30 seconds, onset/offset thresholds) is tuned for speech-dominant audio; long-form noisy recordings deserve their own threshold sweep. Whisper processes audio in fixed 30-second windows regardless of content, which is exactly why the VAD-first design is faster — but it also means a bad VAD run poisons the whole batch. Upstream offers two soft mitigations inside the decoder: --hotwords biases recognition toward rare or technical terms you expect in the audio, and --interleaved_context re-runs the first batch of streams with wrap-around context so chunk boundaries see their neighbors’ tokens.

Diarization needs that HF token and downloads a gated pyannote checkpoint on first run; offline deployments must pre-cache it. And while the batched decoder is dramatically faster than sequential decoding, batch_size is bounded by VRAM: 16 GB comfortably handles large-v2 at batch 16 in float16, less does not.

For readers comparing engines: the batching philosophy here rhymes with what FasterTransformer did for transformer serving, while the model-to-runtime split (CTranslate2 for Whisper, HF transformers auto model loading for wav2vec2, pyannote for speakers) is a reminder that production speech systems are rarely one framework — much like Transformers serves as the common loader layer across the ecosystem. WhisperX’s real lesson is architectural: keep each model in its lane, pass one canonical buffer, and spend your complexity budget on the handoffs.

Enjoyed this post? Never miss out on future posts by following us