Audio Representations & Spectrograms
Audio is a wave — a one-dimensional signal of air pressure over time. Feeding a raw waveform to a model is possible but hard: it's enormously long (tens of thousands of samples per second) and the information you care about lives in its frequencies, not its raw amplitudes. The key move in audio ML is to transform sound into a spectrogram (a 2-D time-frequency image), which turns audio into something CNNs and transformers handle naturally. This lesson covers that pipeline. It's the data stage of the ML project loop for sound: the representation you feed the model (waveform vs mel spectrogram vs MFCC) decides what's even learnable, before any architecture choice.
Real-world examples: every speech recognizer, music classifier, voice assistant, sound-event detector, and audio generation model starts by computing spectrogram-style features (or learns an equivalent front-end).
Walk the whole pipeline once before reading the sections below. Each step says what it produces and what was broken without it, and every number on screen is computed live, so the window-length control lets you test the time–frequency trade-off rather than take it on faith.
From sound to tensor: the audio front-end, step by step
Sound arrives as one very long list of numbers
A microphone turns air pressure into a voltage and the converter reads it 16,000 times a second — the speech convention. One second is 16,000 numbers, and Nyquist caps what those numbers can represent at f_s/2 = 8,000 Hz. Sample too slowly and frequencies above the cap don’t vanish, they alias — they come back disguised as low ones.
This is the honest representation and a poor input. It is enormously long, and the part that carries meaning — which frequencies are present — is nowhere in any single number. It only exists as a relationship between them. Everything that follows is about dragging that relationship out into the open.
Zoom in: the smooth curve is really a list of samples 62.5 µs apart.
waveform
16,000 Hz · 640 ms → 10,240 samples. Nyquist ceiling 8,000 Hz.
A synthetic utterance — two vowels around a fricative — carried through the whole feature pipeline. Every number on screen is computed live in the browser: real Hann windowing, a real radix-2 FFT, a real mel filterbank and a real DCT. Changing the window length re-runs all of it, so the time–frequency trade-off is something you can test rather than something you have to take on faith.
The waveform and sampling
A microphone records a continuous pressure wave; a computer stores samples of it at a fixed sampling rate (e.g. 16 kHz for speech, 44.1 kHz for music). The Nyquist theorem says you can faithfully represent frequencies only up to , so 16 kHz audio captures content up to 8 kHz, enough for intelligible speech.
The result is a long 1-D array: one second of 16 kHz audio is 16,000 numbers. Raw, this is unwieldy and frequency information is implicit.
From time to frequency: the Fourier transform
The Fourier transform rewrites a signal as a sum of sinusoids — revealing which frequencies are present and how strongly. A pure 440 Hz tone is a flat waveform but a single spike in the frequency domain. For audio, frequency content is exactly the perceptually and linguistically meaningful part (pitch, formants, timbre).
But a single Fourier transform over a whole clip loses when each frequency occurred. Speech changes constantly, so we need both time and frequency.
The spectrogram (STFT)
The Short-Time Fourier Transform (STFT) solves this: slide a short window across the signal and take the Fourier transform of each window. The result is a 2-D map (time on one axis, frequency on the other, magnitude as intensity): the spectrogram.
The time–frequency resolution trade-off
The window length sets a fundamental trade-off: a short window localizes events precisely in time but resolves frequency poorly; a long window resolves frequency finely but blurs timing. You can't have perfect resolution in both (a signal-processing uncertainty principle). Speech systems pick a window around 25 ms with a 10 ms hop as a practical balance.
A spectrogram is an image, which is why CNNs (and now transformers) are the natural models for audio — the whole 2-D toolkit transfers directly.
Mel spectrograms and MFCCs
Human hearing isn't linear in frequency: we distinguish low frequencies far more finely than high ones. Two perceptually-motivated transforms make features that match how we hear:
- Mel spectrogram: warp the frequency axis onto the mel scale (roughly logarithmic), so the representation devotes resolution where the ear does. This is the dominant input feature for modern deep speech and audio models.
- MFCC (Mel-Frequency Cepstral Coefficients): apply a further transform (a discrete cosine transform of the log-mel energies) to get a compact, decorrelated feature vector. MFCCs were the feature for classical (pre-deep-learning) speech recognition and remain common for lightweight systems.
| Representation | What it is | Used by |
|---|---|---|
| Raw waveform | 1-D samples | end-to-end models (wav2vec) |
| Spectrogram (STFT) | time × linear-frequency | general audio |
| Mel spectrogram | time × mel-frequency | most deep speech/audio models |
| MFCC | compact cepstral coefficients | classical ASR, small models |
Learned front-ends
Modern systems sometimes skip hand-designed features and learn the front-end from the raw waveform (e.g. wav2vec 2.0's convolutional encoder). But mel spectrograms remain a strong, cheap default — even Whisper uses a log-mel spectrogram as its input.
Exercise
Audio is sampled at 16 kHz. By the Nyquist theorem, what is the highest frequency (in kHz) it can faithfully represent?
Need a hint?
Half the sampling rate.
Exercise
Why is a spectrogram (from the Short-Time Fourier Transform) preferred over a single Fourier transform of the whole audio clip?
Need a hint?
Speech changes over time — what does a single FFT lose?
Exercise
Why do most deep speech models use a mel spectrogram rather than a linear-frequency spectrogram?
Need a hint?
How does human hearing resolve low vs high frequencies?
Common mistakes
- Feeding raw waveforms by default. It's long and frequency information is implicit; spectrogram features are usually easier and cheaper to learn from.
- Ignoring the sampling rate / Nyquist limit. Mismatched sample rates or under-sampling discards or aliases the frequencies you need.
- Wrong window length. Too short blurs frequency, too long blurs time; tune the STFT window to the signal.
- Using a linear frequency axis for speech. The mel scale matches human hearing and consistently works better for speech.
Key takeaways
- Audio is a 1-D waveform sampled at rate ; the Nyquist limit caps representable frequency at .
- The STFT turns a waveform into a 2-D spectrogram (time × frequency), making audio an image that CNNs/transformers handle — subject to a time–frequency resolution trade-off.
- Mel spectrograms warp frequency to match human hearing and are the dominant deep-audio feature; MFCCs are a compact classical variant.
- Some modern models learn the front-end from raw audio, but mel spectrograms remain a strong default (e.g. Whisper).
Related concepts
- Speech Recognition: turning these features into text with CTC and seq2seq models
- Convolution Operation: the 2-D convolution applied to spectrogram images
- Probability Distributions: the signals and spectra underlying frequency analysis