Learning path
Module 29
Speech and audio
Turn sound into model input and text: spectrograms, mel filterbanks, MFCCs, CTC, transducers, attention decoders, and word error rate.
- 1. Spectrograms and mel featuresTurn a waveform into the log-mel spectrogram that speech models consume (framing, windowing, FFT, mel filterbank, log, MFCC) and get the shapes and parameters right.40 min
- 2. Speech recognition: CTC, transducers, and attentionExplain how speech recognizers align audio frames to text (CTC, RNN-T, attention encoder-decoder), decode CTC output greedily and with prefix beam search, compute the CTC loss with the forward algorithm, and measure word error rate.45 min
Audio arrives as tens of thousands of samples per second. Speech models first turn it into a log-mel spectrogram, a compact time-by-frequency image, then align those frames to a much shorter sequence of text tokens. This module builds both halves from scratch in NumPy and connects them to the libraries and pretrained models you would use in practice.
| Stage | What happens | Failure mode |
|---|---|---|
| Resample | bring audio to the model's rate (often 16 kHz) | wrong rate, aliasing |
| Frame and window | 25 ms slices every 10 ms, tapered | leakage, wrong framing |
| FFT and mel | power per frequency, pooled into mel bands | mismatched filterbank or log |
| Encode | convolution and transformer layers | too much downsampling for CTC |
| Align and decode | CTC, transducer, or attention decoder | dropped double letters, hallucination |
| Evaluate | word error rate after normalization | incomparable numbers |
| # | Lesson | You will be able to |
|---|---|---|
| 1 | Spectrograms and mel features | compute frames, STFT, mel filterbank, log-mel, and MFCC with correct shapes, and avoid front-end mismatches |
| 2 | Speech recognition: CTC, transducers, and attention | decode CTC output, compute CTC loss with the forward algorithm, measure WER, and choose an ASR architecture |