Speech Recognition
Automatic Speech Recognition (ASR) maps an audio signal to the words that were spoken. The central difficulty isn't recognizing sounds — it's alignment: a one-second utterance is ~100 spectrogram frames but maybe 3 words, and you don't know which frames correspond to which characters. This lesson covers the two ideas that solved alignment (CTC and attention-based sequence-to-sequence) and how modern systems like Whisper combine them.
Real-world examples: voice assistants, dictation, video captioning and subtitles, call-center transcription, and the speech front-end of multimodal assistants.
The alignment problem
The input is a sequence of audio frames (e.g. 100/second); the output is a much shorter sequence of characters or tokens. Crucially, the alignment between them is unknown and variable: people speak at different speeds, pause, and stretch sounds. Classical systems used hand-built HMM alignments (and pronunciation dictionaries and separate language models) — powerful but complex. The deep-learning era replaced that pipeline with end-to-end models that learn alignment implicitly.
CTC: alignment without labels
Connectionist Temporal Classification (CTC) lets a model output a character per frame and sum over all possible alignments to the target text — no frame-level labels needed. Two ingredients:
- A special blank token the model can emit when it's "between" characters or unsure.
- A collapse rule: merge repeated characters, then remove blanks. So
h h ∅ e ∅ l l ∅ l o→hello.
Given the audio input (the sequence of spectrogram frames), CTC defines the probability of a target as the sum over every frame-labeling that collapses to it:
This sum has exponentially many alignments but is computed efficiently with a dynamic-programming (forward–backward) algorithm — the same DP family as HMMs. CTC assumes outputs are conditionally independent across frames given the audio, which is its main limitation (it leans on an external language model to fix that).
Why the blank token is the key trick
Without a blank, the collapse rule couldn't tell "hello" (one l-sound held long) from "helo" — repeated real characters would always merge. The blank lets the model separate genuine repeats ("ll") from a single sound spanning many frames, making the alignment well-defined.
Sequence-to-sequence and attention
The alternative: treat ASR as translation from audio to text. An encoder processes the spectrogram into a representation; a decoder generates the transcript token by token, using attention to look at the relevant audio frames for each output token.
- Listen, Attend and Spell (LAS) was the canonical attention-based ASR model.
- Unlike CTC, the decoder is autoregressive (each token conditions on the previous ones), so it models the language internally (no conditional-independence assumption) and needs no separate language model.
- The cost: it must be run sequentially and can "hallucinate" fluent-but-wrong text if attention drifts.
| CTC | Attention seq2seq | |
|---|---|---|
| Alignment | monotonic, summed over | learned by attention (flexible) |
| Output dependence | conditionally independent per frame | autoregressive (models language) |
| Decoding | fast, parallel-ish | sequential |
| Needs external LM | helps a lot | optional |
Modern ASR: Whisper and the transformer era
Whisper (and similar models) are encoder–decoder transformers trained on enormous, weakly-labeled audio–text data scraped from the web. The recipe:
- Input a log-mel spectrogram (Lesson 1) to a transformer encoder.
- A transformer decoder generates text autoregressively with cross-attention to the audio — the seq2seq approach, scaled up.
- Massive, diverse training data (multilingual, noisy, many domains) makes it robust zero-shot — it transcribes accents, background noise, and languages it was never specifically tuned for, and even does translation and timestamps.
The lesson of Whisper mirrors the rest of modern ML: a fairly standard architecture plus huge weakly-supervised data beats carefully engineered pipelines. Some production systems still prefer CTC/transducer models for streaming (low-latency, word-by-word) recognition, where an autoregressive decoder over the whole utterance is awkward.
RNN-Transducer: the streaming workhorse
The RNN-T (transducer) combines CTC's frame-synchronous, streaming-friendly nature with an internal language model (it conditions on previous outputs). It's the backbone of many on-device, real-time assistants where you need transcription to appear as you speak, which a full-utterance attention decoder can't do.
Evaluating ASR: Word Error Rate
The standard metric is Word Error Rate (WER) — the edit distance between the reference transcript and the hypothesis, at the word level, normalised by reference length:
where = substitutions (wrong word), = deletions (missing word), = insertions (extra word), and = total words in the reference. A WER of 0 is perfect; values above 1 are possible (many insertions). Human-level conversational WER is roughly 5–8 %; Whisper's multilingual median is ≈ 10–15 %.
Worked example. Reference: "the cat sat on the mat" (). Hypothesis: "the cat sat on a mat hat". Aligning them: "the"→"a" is one substitution (), the trailing "hat" is one insertion (), nothing was deleted ():
Character Error Rate (CER) is the same metric computed at the character level — preferred for languages without clear word boundaries (Chinese, Japanese, Thai) or for measuring fine-grained accuracy in morphologically complex languages.
Every piece above (blank, collapse, the sum over alignments, then WER) is easy to state and hard to feel working together. Walk the whole pipeline below on one utterance of 102 frames: edit a frame labelling and watch it collapse, see the forward DP's probability mass (checked against brute-force enumeration of every labelling), then decode the same acoustic model three ways. Two characters are deliberately barely voiced, and the two errors that causes turn out to need completely different fixes — one is the decoder's fault and one is the model's.
From frames to Word Error Rate, step by step
There are far more frames than characters, and no map between them
The acoustic model emits a distribution over characters for every frame — about 100 a second. The transcript is far shorter, and nobody labelled which frames belong to which character: people stretch vowels, pause, and run words together.
Without a way to handle that, you would need frame-level annotations to train at all, which is exactly the hand-built alignment stage that classical HMM systems needed and CTC removed.
The bright band is the model's per-frame belief. Notice it is a staircase, not a diagonal — each character is held for several frames.
the utterance
102 frames for 22 characters — and 7.9e+33 ways to line them up.
One utterance — 102 frames of character posteriors for six words — run through the whole CTC pipeline in the browser: the forward DP (checked against brute-force enumeration), greedy decoding, prefix beam search, and a lexicon language model, scored with a real Levenshtein alignment. Two characters are deliberately barely voiced, and the two errors that causes have completely different fixes. The articulation and beam-width controls re-run everything.
Exercise
What is the central difficulty (the 'alignment problem') in speech recognition?
Need a hint?
Compare the number of audio frames to the number of output characters.
Exercise
In CTC, what is the role of the special 'blank' token together with the collapse rule (merge repeats, then drop blanks)?
Need a hint?
How do you tell 'hello' (real double-l) from 'helo'?
Exercise
Whisper achieves robust zero-shot speech recognition across accents, noise, and languages. What is the primary reason?
Need a hint?
Architecture novelty, or training data?
Common mistakes
- Forgetting CTC's blank/collapse rule. Without it, repeated characters and held sounds are indistinguishable.
- Assuming attention ASR can stream. A full-utterance autoregressive decoder needs the whole input; use CTC/RNN-T for low latency.
- Ignoring the language model. CTC's per-frame independence makes an external/internal LM important for fluent output.
- Underestimating data. Whisper's robustness comes from scale and diversity of training data more than architectural novelty.
Key takeaways
- ASR's core challenge is alignment: many audio frames map to few output tokens, with unknown, variable correspondence.
- CTC outputs a token per frame with a blank and a collapse rule, summing over all alignments via dynamic programming — fast and streaming-friendly, but assumes per-frame independence.
- Attention seq2seq treats ASR as audio→text translation with an autoregressive decoder that models language internally — flexible but sequential.
- Whisper-style encoder–decoder transformers trained on massive weakly-labeled data give robust zero-shot ASR; RNN-T/CTC remain preferred for streaming.
Related concepts
- Audio Representations & Spectrograms: the log-mel features that feed every ASR model
- Hidden Markov Models: the classical alignment model and the DP that CTC echoes
- Transformer Architecture: the encoder–decoder backbone of modern ASR