Spectrograms and mel features

1. Intuition

A waveform is the wrong shape for learning. One second of audio at 16 kHz is 16,000 numbers between −1 and 1. The same vowel spoken twice gives completely different sample values (shifted by a millisecond, a bit louder), yet sounds the same. What stays stable is which frequencies carry energy, over windows of tens of milliseconds.

From waveform to spectrogram.

1. Frame. Cut the signal into short overlapping frames. The speech default is a 25 ms window every 10 ms: 400 samples with a hop of 160 at 16 kHz. Speech is roughly stable over 25 ms, and 10 ms steps track how fast it changes.
2. Window. Multiply each frame by a bell-shaped Hann window that fades to zero at both ends. Without it, the cut edges look like sudden jumps, and the FFT smears energy across many frequencies ("spectral leakage").
3. FFT. The Fourier transform of each frame says how much of each frequency it contains. Keep the magnitude (or its square, the power).
4. Stack. Frames become columns of a time × frequency image: the spectrogram.

The mel scale. Hearing is roughly logarithmic in pitch: 100 Hz to 200 Hz sounds like a big step; 7,000 Hz to 7,100 Hz barely registers. The mel scale warps frequency to match. A mel filterbank is a set of overlapping triangular filters, equally spaced in mel, so they're narrow at low frequencies and wide at high ones. Each filter adds up the power in its range. 257 FFT bins become 80 mel bands: a mel spectrogram.

Log. Loudness perception is also roughly logarithmic, and raw power spans many orders of magnitude. Taking the log compresses that range so quiet consonants and loud vowels are both visible to the model.

MFCC. Applying a discrete cosine transform (DCT) to each log-mel frame and keeping the first 13 coefficients gives mel-frequency cepstral coefficients (MFCCs). They were the standard feature for decades because they're compact and decorrelated, which suited Gaussian models. Neural networks do fine on log-mels directly, so modern ASR uses log-mels; MFCCs survive in small, classical, and some speaker-recognition systems.

Note

Not every model uses mels. Self-supervised models such as wav2vec 2.0 and HuBERT learn their own front end from the raw waveform with a convolutional encoder. Neural audio codecs (used in many TTS and audio-LLM systems) also start from raw samples. But the log-mel spectrogram is still the most common input, and the concepts (frames, hops, frequency resolution) apply everywhere.

2. Visualization

Synthetic signal: a chirp sweeping 200 to 4,000 Hz for 0.5 s, a 2 kHz tone until 0.9 s, then silence, sampled at 16 kHz.

Compare the two spectrograms. In hertz, the chirp is a straight line because its frequency rises linearly. In mels it bends: low frequencies get many mel bands, so the curve climbs fast at first, then high frequencies share fewer, wider bands, so it flattens. The vertical streaks at 0.5 s and 0.9 s are real: an abrupt change in a signal contains energy at all frequencies, which is also why clicks and plosives ("p", "t") look like vertical lines in speech spectrograms.

Try it (predict first, then edit mel_features.py):

1. Set WIN = 1600 (100 ms). Predict what happens to the chirp line's thickness and to the sharpness of the 0.5 s transition.
2. Replace hann(...) with a window of all ones (no tapering). Predict the change in the tone's spectrogram row.
3. Set N_MELS = 128. Predict how many low filters cover just one FFT bin, and whether the chirp curve looks smoother.
4. Resample conceptually: change SR to 8,000 while keeping the chirp to 4,000 Hz. Predict what happens to the top of the sweep. (Hint: the highest frequency a sampled signal can represent is half the sample rate.)

3. The math

Symbols

Framing

Frame $t$ covers samples $tH$ to $tH + W - 1$.

Window and FFT

Hann window (periodic form, as in PyTorch and librosa): $w[n] = 0.5 - 0.5\cos(2\pi n / W)$. Power spectrum of frame $t$ at bin $k$:

Bin $k$ corresponds to frequency $k f_s / N_{\text{fft}}$. Only bins up to $N_{\text{fft}}/2$ are kept because a real signal's spectrum is symmetric.

Mel scale and filterbank

Place $M + 2$ edges equally spaced in mel from $m(f_{\min})$ to $m(f_{\max})$, convert back to hertz as $e_0, \dots, e_{M+1}$. Filter $j$ is a triangle rising from $e_j$ to a peak of 1 at $e_{j+1}$ and falling to 0 at $e_{j+2}$. For a bin at frequency $f$:

Then the whole mel step is one matrix multiply and a log:

Worked example

Settings: $f_s = 16{,}000$, $L = 16{,}000$ (1 s), $W = 400$, $H = 160$, $N_{\text{fft}} = 512$, $M = 80$.

1. Frames: $F = 1 + \lfloor (16{,}000 - 400)/160 \rfloor = 1 + \lfloor 15{,}600 / 160 \rfloor = 1 + 97 = 98$. Frame shape $98 \times 400$.
2. Bins: $512 / 2 + 1 = 257$ bins, spaced $16{,}000 / 512 = 31.25$ Hz. Power spectrogram shape $98 \times 257$.
3. A 2 kHz tone lands at bin $2{,}000 / 31.25 = 64$ exactly.
4. Mel range: $m(8{,}000) = 2595 \log_{10}(1 + 8{,}000/700) = 2595 \log_{10}(12.4286) = 2595 \times 1.09442 = 2840.0$ mel. 82 edges from 0 to 2840.0 are $2840.0 / 81 = 35.06$ mel apart.
5. Where 2 kHz falls: $m(2{,}000) = 2595 \log_{10}(3.8571) = 2595 \times 0.58627 = 1521.4$ mel. Edges 43 and 44 are at $43 \times 35.06 = 1507.7$ and $44 \times 35.06 = 1542.7$ mel, which convert to $e_{43} = 1967.4$ Hz and $e_{44} = 2051.7$ Hz.
6. Weights for bin 64 (2,000 Hz): filter 42 (edges 42, 43, 44) is on its falling side: $(2051.7 - 2000)/(2051.7 - 1967.4) = 51.7 / 84.3 = 0.613$. Filter 43 (edges 43, 44, 45) is on its rising side: $(2000 - 1967.4)/84.3 = 32.6 / 84.3 = 0.387$. The two weights sum to 1, because adjacent triangles overlap exactly.
7. Shapes: $B$ is $80 \times 257$; $P B^\top$ is $(98 \times 257)(257 \times 80) = 98 \times 80$; MFCCs are $98 \times 13$.
8. Filter widths: filter 0 spans 0 to 45 Hz, so with bins 31.25 Hz apart it contains a single FFT bin (31.25 Hz). Filter 79 spans 7,475 to 8,000 Hz (525 Hz). Equal steps in mel are very unequal in hertz.

Time versus frequency resolution. Bin spacing is $f_s / N_{\text{fft}}$, and each frame spans $W / f_s$ seconds. A longer window resolves nearby pitches but blurs fast events; a shorter one does the opposite. 25 ms / 10 ms is the speech compromise.

4. Implementation

Build it from scratch

All of it is NumPy. First, framing. Instead of a Python loop, build a $F \times W$ index matrix: row $t$ is $tH, tH+1, \dots, tH+W-1$, and fancy indexing pulls every frame at once.

def frame(x, win=WIN, hop=HOP):
    n = 1 + (len(x) - win) // hop
    idx = np.arange(win)[None, :] + hop * np.arange(n)[:, None]           # [n_frames, win]
    return x[idx]

Predict: a 0.5 s clip at 16 kHz with the same window and hop gives how many frames?

Window and FFT. np.fft.rfft returns only the non-negative frequencies, and n=N_FFT zero-pads each 400-sample frame to 512.

def hann(n):
    return 0.5 - 0.5 * np.cos(2 * np.pi * np.arange(n) / n)              # periodic Hann (as torch/librosa)

def power_spectrum(frames, n_fft=N_FFT):
    spec = np.fft.rfft(frames * hann(frames.shape[1]), n=n_fft)           # [n_frames, n_fft//2 + 1]
    return np.abs(spec) ** 2

Predict: zero-padding to 512 makes the bins closer together (31.25 Hz instead of 40 Hz). Does it let you separate two tones 35 Hz apart? (No: padding interpolates the spectrum; true resolution is set by the 25 ms window.)

The mel conversion and the filterbank. Each row of fb is one triangle evaluated at every FFT bin's frequency.

def mel_filterbank(n_mels=N_MELS, n_fft=N_FFT, sr=SR, fmin=0.0, fmax=None):
    fmax = fmax or sr / 2
    edges_hz = mel_to_hz(np.linspace(hz_to_mel(fmin), hz_to_mel(fmax), n_mels + 2))  # equally spaced in mel
    bin_hz = np.arange(n_fft // 2 + 1) * sr / n_fft                                    # centre of each FFT bin
    fb = np.zeros((n_mels, len(bin_hz)))
    for m in range(n_mels):
        lo, c, hi = edges_hz[m], edges_hz[m + 1], edges_hz[m + 2]
        rise = (bin_hz - lo) / (c - lo)
        fall = (hi - bin_hz) / (hi - c)
        fb[m] = np.maximum(0, np.minimum(rise, fall))                     # triangle peaking at 1 at c
    return fb, edges_hz

Predict: what is the shape of fb for 80 mels and n_fft=512, and roughly how many nonzero entries does row 79 have? (Script: $80 \times 257$, 17.)

Log-mel is one matmul and a log; MFCC is a DCT-II written as another matmul with a cosine basis.

def log_mel(power, fb, eps=1e-10):
    return np.log10(np.maximum(power @ fb.T, eps))                      # [n_frames, n_mels]

def dct_ii(x, n_coeffs=13):
    N = x.shape[-1]
    n = np.arange(N)
    basis = np.cos(np.pi * np.arange(n_coeffs)[:, None] * (2 * n + 1) / (2 * N))   # [n_coeffs, N]
    return x @ basis.T

Predict: in a silent frame, what value does every log-mel entry take? (The floor: $\log_{10}(10^{-10}) = -10$. That's why the floor value matters.)

Use the library

The same pipeline in one call each:

import torchaudio, librosa

mel = torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_fft=400, win_length=400,
                                           hop_length=160, n_mels=80)(waveform)  # [channels, 80, frames]
log_mel = torch.log10(mel.clamp(min=1e-10))

S = librosa.feature.melspectrogram(y=y, sr=16000, n_fft=400, hop_length=160, n_mels=80)
mfcc = librosa.feature.mfcc(S=librosa.power_to_db(S), n_mfcc=13)

Note the conventions that differ from our script and from each other: libraries put time on the last axis ($80 \times F$, not $F \times 80$); librosa's default mel scale is the Slaney variant (linear below 1 kHz) with area-normalized filters, while torchaudio's default is the HTK formula used here; power_to_db gives $10 \log_{10}$ relative to a reference. Whisper computes 25 ms / 10 ms frames at 16 kHz, applies a mel filterbank (80 bands in most sizes, 128 in large-v3), takes $\log_{10}$, clips to 8 below the maximum, and rescales. If you feed a pretrained model, use its own feature extractor (for example the Hugging Face WhisperFeatureExtractor), never a re-implementation.

Run it

cd guide
uv run code/29-speech-and-audio/mel_features.py

Key output:

frames: 1 + (16000 - 400) // 160 = 98  -> shape (98, 400)
power spectrum shape (98, 257); bin spacing 31.25 Hz
frame 70 (t = 0.70 s): peak bin 64 -> 2000.0 Hz (true 2000 Hz)
filterbank shape (80, 257); filter 0 spans 0-45 Hz (45 Hz wide), filter 79 spans 7475-8000 Hz (525 Hz wide)
log-mel shape (98, 80); MFCC shape (98, 13)
  frame 25 (t = 0.263 s): mel 45, centre   2228 Hz, true   2195 Hz
silent frame 95: max log-mel -10.0 (floor log10(1e-10) = -10)

Full script: code/29-speech-and-audio/mel_features.py.

5. Engineering

Typical settings.

Sample rate is part of the model. A model trained at 16 kHz fed 44.1 kHz audio sees every frequency scaled wrong. Resample at the input boundary, with a proper anti-aliasing filter (torchaudio.functional.resample, librosa.resample), and assert the rate in code.

Normalization must match training. Per-utterance mean/variance normalization, global statistics, or Whisper-style clipping and rescaling all produce different numbers. A mismatch can silently degrade accuracy. Ship the feature extractor with the model and test that features match on a golden file.

Padding and length. Whisper pads or trims every input to 30 s (3,000 frames). Streaming models need features computed incrementally with the same framing as offline. Zero-padding produces frames at the log floor; make sure the model saw the same during training, or pass an attention mask.

Compute. The front end is cheap next to the model (an FFT per 10 ms frame), but on small devices it matters: precompute the filterbank once, use real FFTs, and run in float32.

Augmentation. SpecAugment (masking random time and frequency bands of the log-mel) is a standard, cheap regularizer for speech models, applied to features during training only.

Common mistakes

• Feeding audio at the wrong sample rate, or resampling with naive decimation that aliases.
• Mixing natural log, $\log_{10}$, and decibels between training and serving.
• Transposing time and mel axes ($F \times 80$ vs $80 \times F$) when moving between libraries.
• Too many mel filters for the FFT size, leaving low filters covering one or zero bins (librosa warns about empty filters).
• Re-implementing a pretrained model's feature extractor "close enough" instead of using its own.
• Forgetting the window, which smears every tone across neighbouring bins.

6. Knowledge check

Practice exercise. A 3-second clip at 16 kHz uses a 25 ms window, 10 ms hop, $N_{\text{fft}} = 400$, and 80 mels. Give the number of samples, frames, FFT bins, bin spacing in hertz, and the shapes of the power spectrogram, filterbank, and log-mel output.

Implementation challenge. Add delta features: for each log-mel frame $t$, compute $\Delta_t = (S_{t+1} - S_{t-1}) / 2$ (edge-replicate the ends), then stack $[S, \Delta]$ into a $F \times 160$ matrix. Check on the chirp that the delta is positive just above the chirp's current band and negative just below it.

Summary

• Speech front ends frame the waveform (25 ms window, 10 ms hop at 16 kHz), apply a Hann window, and take an FFT per frame to get a power spectrogram.
• The mel filterbank is an $M \times (N_{\text{fft}}/2+1)$ matrix of triangles equally spaced in mel; one matrix multiply turns 257 bins into 80 bands.
• Log compresses dynamic range; MFCCs add a DCT and keep about 13 coefficients, now mostly replaced by log-mels for neural models.
• Window length trades time resolution against frequency resolution.
• Sample rate, log base, normalization, axis order, and padding must match what the model was trained on.

Next: Speech recognition: CTC, transducers, and attention

Related: Convolutional networks · Recurrent networks · Transformer architecture

Go deeper, code-first (AI Engineering from Scratch): The Fourier Transform · Audio Fundamentals — Waveforms, Sampling, Fourier Transform · Audio Classification — From k-NN on MFCCs to AST and BEATs

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Spectrograms, Mel Scale & Audio Features.

Interview angle