Speech RecognitionOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Speech Recognition

Automatic Speech Recognition (ASR) maps an audio signal to the words that were spoken. The central difficulty isn't recognizing sounds — it's alignment: a one-second utterance is ~100 spectrogram frames but maybe 3 words, and you don't know which frames correspond to which characters. This lesson covers the two ideas that solved alignment (CTC and attention-based sequence-to-sequence) and how modern systems like Whisper combine them.

Real-world examples: voice assistants, dictation, video captioning and subtitles, call-center transcription, and the speech front-end of multimodal assistants.

The alignment problem

The input is a sequence of TT audio frames (e.g. 100/second); the output is a much shorter sequence of UU characters or tokens. Crucially, the alignment between them is unknown and variable: people speak at different speeds, pause, and stretch sounds. Classical systems used hand-built HMM alignments (and pronunciation dictionaries and separate language models) — powerful but complex. The deep-learning era replaced that pipeline with end-to-end models that learn alignment implicitly.

CTC: alignment without labels

Connectionist Temporal Classification (CTC) lets a model output a character per frame and sum over all possible alignments to the target text — no frame-level labels needed. Two ingredients:

  • A special blank token ∅\varnothing the model can emit when it's "between" characters or unsure.
  • A collapse rule: merge repeated characters, then remove blanks. So h h ∅ e ∅ l l ∅ l o → hello.

Given the audio input XX (the sequence of spectrogram frames), CTC defines the probability of a target as the sum over every frame-labeling that collapses to it:

P("hello"∣X)=∑π ∈ align(hello)∏tP(πt∣X)P(\text{"hello"}\mid X) = \sum_{\pi \,\in\, \text{align(hello)}} \prod_{t} P(\pi_t \mid X)

This sum has exponentially many alignments but is computed efficiently with a dynamic-programming (forward–backward) algorithm — the same DP family as HMMs. CTC assumes outputs are conditionally independent across frames given the audio, which is its main limitation (it leans on an external language model to fix that).

Why the blank token is the key trick

Without a blank, the collapse rule couldn't tell "hello" (one l-sound held long) from "helo" — repeated real characters would always merge. The blank lets the model separate genuine repeats ("ll") from a single sound spanning many frames, making the alignment well-defined.

Sequence-to-sequence and attention

The alternative: treat ASR as translation from audio to text. An encoder processes the spectrogram into a representation; a decoder generates the transcript token by token, using attention to look at the relevant audio frames for each output token.

  • Listen, Attend and Spell (LAS) was the canonical attention-based ASR model.
  • Unlike CTC, the decoder is autoregressive (each token conditions on the previous ones), so it models the language internally (no conditional-independence assumption) and needs no separate language model.
  • The cost: it must be run sequentially and can "hallucinate" fluent-but-wrong text if attention drifts.
CTCAttention seq2seq
Alignmentmonotonic, summed overlearned by attention (flexible)
Output dependenceconditionally independent per frameautoregressive (models language)
Decodingfast, parallel-ishsequential
Needs external LMhelps a lotoptional

Modern ASR: Whisper and the transformer era

Whisper (and similar models) are encoder–decoder transformers trained on enormous, weakly-labeled audio–text data scraped from the web. The recipe:

  1. Input a log-mel spectrogram (Lesson 1) to a transformer encoder.
  2. A transformer decoder generates text autoregressively with cross-attention to the audio — the seq2seq approach, scaled up.
  3. Massive, diverse training data (multilingual, noisy, many domains) makes it robust zero-shot — it transcribes accents, background noise, and languages it was never specifically tuned for, and even does translation and timestamps.

The lesson of Whisper mirrors the rest of modern ML: a fairly standard architecture plus huge weakly-supervised data beats carefully engineered pipelines. Some production systems still prefer CTC/transducer models for streaming (low-latency, word-by-word) recognition, where an autoregressive decoder over the whole utterance is awkward.

RNN-Transducer: the streaming workhorse

The RNN-T (transducer) combines CTC's frame-synchronous, streaming-friendly nature with an internal language model (it conditions on previous outputs). It's the backbone of many on-device, real-time assistants where you need transcription to appear as you speak, which a full-utterance attention decoder can't do.

Evaluating ASR: Word Error Rate

The standard metric is Word Error Rate (WER) — the edit distance between the reference transcript and the hypothesis, at the word level, normalised by reference length:

WER=S+D+IN\text{WER} = \frac{S + D + I}{N}

where SS = substitutions (wrong word), DD = deletions (missing word), II = insertions (extra word), and NN = total words in the reference. A WER of 0 is perfect; values above 1 are possible (many insertions). Human-level conversational WER is roughly 5–8 %; Whisper's multilingual median is ≈ 10–15 %.

Worked example. Reference: "the cat sat on the mat" (N=6N = 6). Hypothesis: "the cat sat on a mat hat". Aligning them: "the"→"a" is one substitution (S=1S=1), the trailing "hat" is one insertion (I=1I=1), nothing was deleted (D=0D=0):

WER=1+0+16=0.333  (33.3%).\text{WER} = \frac{1 + 0 + 1}{6} = 0.333 \;(33.3\%).

Character Error Rate (CER) is the same metric computed at the character level — preferred for languages without clear word boundaries (Chinese, Japanese, Thai) or for measuring fine-grained accuracy in morphologically complex languages.

Every piece above (blank, collapse, the sum over alignments, then WER) is easy to state and hard to feel working together. Walk the whole pipeline below on one utterance of 102 frames: edit a frame labelling and watch it collapse, see the forward DP's probability mass (checked against brute-force enumeration of every labelling), then decode the same acoustic model three ways. Two characters are deliberately barely voiced, and the two errors that causes turn out to need completely different fixes — one is the decoder's fault and one is the model's.

From frames to Word Error Rate, step by step

The alignment problemstep 1 of 7
There are far more frames than characters, and no map between them

The acoustic model emits a distribution over characters for every frame — about 100 a second. The transcript is far shorter, and nobody labelled which frames belong to which character: people stretch vowels, pause, and run words together.

Without a way to handle that, you would need frame-level annotations to train at all, which is exactly the hand-built alignment stage that classical HMM systems needed and CTC removed.

The bright band is the model's per-frame belief. Notice it is a staircase, not a diagonal — each character is held for several frames.

articulationbeam width
Frames vs texttwo barely-voiced characters · beam width 40
_␣acehkmnost102 frames · 22 characters · 6 words
the utterance

102 frames for 22 characters — and 7.9e+33 ways to line them up.

per-frame posteriorbarely-voiced frames

One utterance — 102 frames of character posteriors for six words — run through the whole CTC pipeline in the browser: the forward DP (checked against brute-force enumeration), greedy decoding, prefix beam search, and a lexicon language model, scored with a real Levenshtein alignment. Two characters are deliberately barely voiced, and the two errors that causes have completely different fixes. The articulation and beam-width controls re-run everything.

Exercise

What is the central difficulty (the 'alignment problem') in speech recognition?

Need a hint?

Compare the number of audio frames to the number of output characters.

Exercise

In CTC, what is the role of the special 'blank' token together with the collapse rule (merge repeats, then drop blanks)?

Need a hint?

How do you tell 'hello' (real double-l) from 'helo'?

Exercise

Whisper achieves robust zero-shot speech recognition across accents, noise, and languages. What is the primary reason?

Need a hint?

Architecture novelty, or training data?

Common mistakes

  1. Forgetting CTC's blank/collapse rule. Without it, repeated characters and held sounds are indistinguishable.
  2. Assuming attention ASR can stream. A full-utterance autoregressive decoder needs the whole input; use CTC/RNN-T for low latency.
  3. Ignoring the language model. CTC's per-frame independence makes an external/internal LM important for fluent output.
  4. Underestimating data. Whisper's robustness comes from scale and diversity of training data more than architectural novelty.

Key takeaways

  • ASR's core challenge is alignment: many audio frames map to few output tokens, with unknown, variable correspondence.
  • CTC outputs a token per frame with a blank and a collapse rule, summing over all alignments via dynamic programming — fast and streaming-friendly, but assumes per-frame independence.
  • Attention seq2seq treats ASR as audio→text translation with an autoregressive decoder that models language internally — flexible but sequential.
  • Whisper-style encoder–decoder transformers trained on massive weakly-labeled data give robust zero-shot ASR; RNN-T/CTC remain preferred for streaming.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Speech RecognitionConvolutional Neural Netw…Recurrent Neural NetworksVoice & Multimodal AIAudio Representations & S…Hidden Markov ModelsThe Transformer Architect…