Speech recognition: CTC, transducers, and attention

1. Intuition

The alignment problem. After the front end, a clip is a sequence of feature frames, 100 per second. An encoder (convolutions, then a transformer or Conformer stack) turns them into $T$ hidden vectors, often downsampled to one per 40 to 80 ms. The transcript is a shorter sequence of $U$ tokens (characters or subwords). We don't know which frames produced which token, and a word can take 150 ms or 1.5 s. Training data has only (audio, transcript) pairs.

CTC (Connectionist Temporal Classification). Add a special blank token meaning "nothing new here". The model outputs a probability distribution over tokens-plus-blank at every frame, independently. Any frame-level path maps to text by a collapse rule: merge consecutive repeats, then delete blanks. So hh_e_ll_lloo becomes "hello": the blank between the two ll groups is what keeps the double letter. Training maximizes the total probability of all paths that collapse to the correct transcript. Decoding is fast because frames are independent. The cost is that independence: CTC can't model that "their" and "there" depend on the surrounding words, so it is often paired with an external language model at decode time.

RNN-T (transducer). Add a small prediction network that reads the tokens emitted so far, and a joint network that combines it with the current audio frame to predict "next token, or move to the next frame". Output now depends on previous outputs, an internal language model of sorts, while still processing audio left to right. That makes transducers the standard choice for streaming and on-device recognition. Training is heavier: the loss sums over a 2D lattice of (frame, token) positions. Token-and-duration transducers (TDT) also predict how many frames to skip, which speeds up decoding.

Attention encoder-decoder. The encoder reads all the audio; a transformer decoder generates tokens one at a time, cross-attending to any part of the encoder output. This is the same autoregressive generation as an LLM (see decoding); Whisper is the best-known example. It gives strong offline accuracy and handles punctuation, casing, and translation naturally. It doesn't stream without extra engineering, latency grows with output length, and like any autoregressive generator it can hallucinate: produce fluent text that wasn't said, especially on silence, music, or noise.

Measuring it: WER. Word error rate counts the minimum number of word substitutions, deletions, and insertions that turn the hypothesis into the reference, divided by the number of reference words. Lower is better, and it can exceed 1.0 because insertions are unbounded.

Key distinction

WER depends on text normalization. "Turn on the lights." and "turn on the lights" differ by casing and punctuation. Published comparisons apply a normalizer (lowercase, strip punctuation, standardize numbers and spellings) before computing WER. Comparing your system's raw WER to a published normalized WER is meaningless.

2. Visualization

Synthetic posteriors (no audio): what an acoustic encoder might output for "hello world". Right: the forward algorithm's table, in log10, restricted to the first 20 frames.

On the left, the orange dots are greedy decoding: take the most likely token at each frame. Reading them in order gives _hhh_eee_lll_lll_ooo_ …, and the collapse rule turns that into "hello world". Note the blank between the two lll blocks: without it, the collapse would merge them into one l. On the right, each column is one frame and each row is a position in _h_e_l_l_o_. A bright cell means "a lot of probability flows through being at this label position at this frame". The diagonal band is the set of plausible alignments; the loss sums them all instead of picking one.

Try it (predict first, then edit ctc_asr.py):

1. In synthetic_posteriors, remove the line that inserts a blank between repeated letters. Predict the greedy output for "hello".
2. Lower peak from 0.8 to 0.3. Predict whether greedy and beam search still agree, and what kinds of errors appear.
3. In the two-frame example, change the probabilities to $P(\_) = 0.45$, $P(a) = 0.55$. Predict the greedy output and $P(\text{"a"})$.
4. Add the hypothesis "turn off the kitchen lights" to the WER list. Predict its WER and the S/D/I split.

3. The math

Symbols

CTC probability and loss

There are $(V+1)^T$ paths, far too many to enumerate. The forward algorithm sums them in $O(T \cdot S)$ time. With $y' = (\_, y_1, \_, y_2, \dots, y_U, \_)$:

A path can stay at the same label, move one position forward, or skip one blank. The skip is allowed only when $y'_s$ is not blank and $y'_s \ne y'_{s-2}$: you can't jump over the blank between two identical letters, because that would merge them. It starts with $\alpha_1(1) = p_1(\_)$, $\alpha_1(2) = p_1(y_1)$, and ends with $P(y \mid x) = \alpha_T(S) + \alpha_T(S-1)$ (finish on the final blank or the final letter). A valid path needs $T \ge U + r$, where $r$ is the number of adjacent repeated letters in $y$, since each repeat needs a blank between.

Worked example: T = 3, vocabulary {blank, a}, target "a"

Frame probabilities ($p_t(\_)$, $p_t(a)$): $(0.4, 0.6)$, $(0.5, 0.5)$, $(0.7, 0.3)$. Extended target $y' = (\_, a, \_)$, $S = 3$.

Brute force. Of the $2^3 = 8$ paths, six collapse to "a": __a $= 0.4 \times 0.5 \times 0.3 = 0.06$; _a_ $= 0.4 \times 0.5 \times 0.7 = 0.14$; _aa $= 0.4 \times 0.5 \times 0.3 = 0.06$; a__ $= 0.6 \times 0.5 \times 0.7 = 0.21$; aa_ $= 0.6 \times 0.5 \times 0.7 = 0.21$; aaa $= 0.6 \times 0.5 \times 0.3 = 0.09$. Sum $= 0.06 + 0.14 + 0.06 + 0.21 + 0.21 + 0.09 = 0.77$. (___ gives "" and a_a gives "aa".)

Forward algorithm.

1. $t = 1$: $\alpha_1 = (0.4, \; 0.6, \; 0)$.
2. $t = 2$, multiply by $(0.5, 0.5, 0.5)$: $\alpha_2(1) = 0.4 \times 0.5 = 0.2$; $\alpha_2(2) = (0.6 + 0.4) \times 0.5 = 0.5$; $\alpha_2(3) = (0 + 0.6) \times 0.5 = 0.3$.
3. $t = 3$, probabilities $(0.7, 0.3, 0.7)$ for $(\_, a, \_)$: $\alpha_3(1) = 0.2 \times 0.7 = 0.14$; $\alpha_3(2) = (0.5 + 0.2) \times 0.3 = 0.21$; $\alpha_3(3) = (0.3 + 0.5) \times 0.7 = 0.56$.
4. $P(\text{"a"}) = \alpha_3(3) + \alpha_3(2) = 0.56 + 0.21 = 0.77$. ✓
5. Loss $= -\ln 0.77 = 0.2614$.

Greedy is not the most probable transcript

Two frames, each with $p(\_) = 0.6$, $p(a) = 0.4$. Greedy picks __, which collapses to "" with probability $0.6 \times 0.6 = 0.36$. But "a" collects three paths: aa $0.16$ + a_ $0.24$ + _a $0.24$ $= 0.64$. The best path is not the best transcript. Prefix beam search fixes this by summing path probabilities per collapsed prefix.

Word error rate

Reference "turn on the kitchen lights" ($N = 5$), hypothesis "turn the kitchen light on": "on" deleted after "turn" ($D = 1$), "lights" → "light" ($S = 1$), "on" inserted at the end ($I = 1$). $\text{WER} = 3/5 = 0.60$. The hypothesis "um so turn uh on the the kitchen lights please" contains every reference word but adds 5 insertions: $\text{WER} = 5/5 = 1.00$.

4. Implementation

Build it from scratch

Start with the collapse rule and greedy decoding. prev tracks the last frame's token, so repeats merge, and blanks are dropped but still reset prev, which is what lets "ll" survive.

def collapse(ids):
    out, prev = [], None
    for i in ids:
        if i != prev and i != BLANK:
            out.append(i)
        prev = i
    return out

def greedy(probs):
    return "".join(VOCAB[i] for i in collapse(probs.argmax(1)))

Predict: what does collapse return for the path a a _ a b b?

The forward algorithm. Build the extended label sequence, then fill the $T \times S$ table using stay, step, and skip.

def ctc_prob(probs, target):
    ext = [BLANK]
    for c in target:
        ext += [c, BLANK]                             # "ab" -> _ a _ b _ ; length 2U + 1
    S, T = len(ext), len(probs)
    alpha = np.zeros((T, S))
    alpha[0, 0] = probs[0, ext[0]]
    alpha[0, 1] = probs[0, ext[1]] if S > 1 else 0.0
    for t in range(1, T):
        for s in range(S):
            a = alpha[t - 1, s] + (alpha[t - 1, s - 1] if s >= 1 else 0.0)
            if s >= 2 and ext[s] != BLANK and ext[s] != ext[s - 2]:
                a += alpha[t - 1, s - 2]              # skip the blank between different letters
            alpha[t, s] = a * probs[t, ext[s]]
    return alpha[-1, -1] + (alpha[-1, -2] if S > 1 else 0.0), alpha

Predict: for the target "abb", which skip transitions are forbidden? (The jump from the first b over the blank to the second b.)

Real implementations work in log space (sums become log-sum-exp) because products of hundreds of probabilities underflow. The script checks the forward algorithm against brute-force enumeration on a random 6-frame case, so you can trust it.

Prefix beam search keeps, for each candidate text prefix, two probabilities: ending in blank and ending in a non-blank. The split is needed to decide whether the next identical letter extends the prefix or merges with it.

def prefix_beam(probs, beam=8):
    beams = {(): (1.0, 0.0)}                        # prefix -> (P ending in blank, P ending in non-blank)
    for p in probs:
        nxt = defaultdict(lambda: [0.0, 0.0])
        for prefix, (pb, pnb) in beams.items():
            nxt[prefix][0] += (pb + pnb) * p[BLANK]                    # emit blank: prefix unchanged
            for c in range(1, len(p)):
                if prefix and prefix[-1] == c:
                    nxt[prefix][1] += pnb * p[c]                       # repeat without blank: merges
                    nxt[prefix + (c,)][1] += pb * p[c]                 # blank in between: new letter
                else:
                    nxt[prefix + (c,)][1] += (pb + pnb) * p[c]
        beams = dict(sorted(nxt.items(), key=lambda kv: -sum(kv[1]))[:beam])
    best = max(beams.items(), key=lambda kv: sum(kv[1]))
    return "".join(VOCAB[i] for i in best[0]), sum(best[1])

Predict: on the two-frame example ($p(\_) = 0.6$, $p(a) = 0.4$), what does prefix_beam return? (Script: "a" with 0.64.) This is where an external language model plugs in: multiply each extension by $P_{\text{LM}}(c \mid \text{prefix})^{\lambda}$ when a word completes.

Word error rate is edit distance over words; the script also backtraces the table to count S, D, and I separately.

def wer(ref, hyp):
    r, h = ref.split(), hyp.split()
    d = np.zeros((len(r) + 1, len(h) + 1), dtype=int)
    d[:, 0], d[0, :] = np.arange(len(r) + 1), np.arange(len(h) + 1)
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            d[i, j] = min(d[i - 1, j] + 1, d[i, j - 1] + 1, d[i - 1, j - 1] + (r[i - 1] != h[j - 1]))

Predict: "the cat sat" versus "the cat sat down". WER? ($1/3 = 0.33$, one insertion.)

Use the library

Training with CTC in PyTorch is one loss call; inputs are log-probabilities shaped time-first.

import torch.nn.functional as F
loss = F.ctc_loss(log_probs,        # [T, batch, V+1], from log_softmax
                  targets,          # concatenated target ids
                  input_lengths, target_lengths, blank=0)

For inference, use a pretrained model:

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3-turbo")
print(asr("clip.wav", return_timestamps=True)["text"])   # long audio is chunked into 30 s windows

Other common choices: faster-whisper (CTranslate2 backend, faster CPU/GPU inference of Whisper models), NVIDIA NeMo (Parakeet and Canary families: transducer, CTC, and attention models; strong streaming support), and pyctcdecode for CTC beam search with an n-gram language model. Compute WER with jiwer after applying the same text normalizer to reference and hypothesis (Whisper ships an EnglishTextNormalizer). Check the Hugging Face Open ASR Leaderboard for current, normalized results on public test sets instead of quoting numbers from memory.

Run it

cd guide
uv run code/29-speech-and-audio/ctc_asr.py

Key output:

  forward algorithm P('a') = 0.770; brute force = 0.770; CTC loss = -ln P = 0.2614
  random check, T = 6, V = 4, target 'abb': forward 0.000370 vs brute force 0.000370
  P('a') = aa + a_ + _a = 0.16 + 0.24 + 0.24 = 0.64
  greedy -> ''; prefix beam -> 'a' with p = 0.64
  posteriors shape (46, 28) (frames x vocab)
  greedy: 'hello world'
  WER 0.60  S=1 D=1 I=1  'turn the kitchen light on'
  WER 1.00  S=0 D=0 I=5  'um so turn uh on the the kitchen lights please'

Full script: code/29-speech-and-audio/ctc_asr.py.

5. Engineering

Choosing an architecture.

Long audio. Whisper-style models take 30 s windows. Split long audio with voice activity detection (VAD) at pauses, transcribe chunks (in parallel batches), and stitch with overlap. Splitting mid-word causes errors at the seams.

Hallucination. Autoregressive decoders can invent text on silence or music, or loop on a phrase. Mitigations: VAD to skip non-speech, compression-ratio and log-probability thresholds to reject degenerate outputs, temperature fallback, and not conditioning on the previous chunk's text when it's wrong.

Language and domain. Force the language when you know it; automatic language ID can misfire on short or noisy clips. Accented speech, telephone audio (8 kHz), far-field microphones, and overlapping speakers all raise WER sharply; evaluate on audio from your real channel, not just clean benchmarks.

Evaluation. Report WER on a held-out set from your domain with a fixed normalizer, plus a breakdown by speaker group, channel, and noise condition. For voice agents, also measure latency (time to first word, time to final transcript) and the effect of recognition errors on the downstream task, since the LLM may be robust to some errors and not others.

Privacy. Voice is biometric and often contains personal data. Decide what audio and transcripts you store, for how long, and whether processing can stay on device.

Common mistakes

• Comparing WER numbers computed with different normalizers or on different test sets.
• Feeding 44.1 kHz or 8 kHz audio to a 16 kHz model without resampling.
• Forgetting that CTC needs $T \ge U + r$: downsampling the encoder too aggressively makes long or repetitive transcripts impossible to emit.
• Naive beam search that merges paths by collapsed text without tracking blank versus non-blank, which breaks double letters.
• Sending hours of audio to a 30-second-window model without chunking and VAD.
• Trusting confident-looking output on silence; attention decoders can produce fluent text from noise.

6. Knowledge check

Practice exercise. Two frames, vocabulary {blank, a, b}, probabilities per frame $(p_\_, p_a, p_b)$: frame 1 $(0.5, 0.3, 0.2)$, frame 2 $(0.4, 0.2, 0.4)$. Compute $P(\text{"a"})$, $P(\text{"b"})$, $P(\text{"ab"})$, and $P(\text{""})$, and give the greedy decode.

Implementation challenge. Convert ctc_prob to log space using np.logaddexp, and verify it matches the probability-space version on the "hello world" posteriors. Then compute the loss for a 200-frame synthetic utterance where the probability-space version underflows to 0.

Summary

• Speech recognition must align many audio frames to fewer tokens without knowing the alignment.
• CTC predicts a token or blank per frame independently, collapses repeats and blanks, and trains on the sum over all valid paths, computed by the forward algorithm in O(T·S).
• Greedy CTC decoding picks the best path, not the best transcript; prefix beam search sums paths per prefix and is where language models plug in.
• Transducers add a prediction network for streaming with output dependencies; attention encoder-decoders like Whisper give strong offline quality but don't stream natively and can hallucinate.
• WER = (S + D + I) / N, computed after a fixed text normalization; it can exceed 1.

Next: Gradient clipping and mixed precision

Related: Spectrograms and mel features · Recurrent networks · Transformer architecture · LLM evaluation

Go deeper, code-first (AI Engineering from Scratch): Whisper — Architecture & Fine-Tuning · Text-to-Speech (TTS) — From Tacotron to F5 and Kokoro · Audio Evaluation — WER, MOS, UTMOS, MMAU, FAD, and the Open Leaderboards

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Speech Recognition (ASR) — CTC, RNN-T, Attention.

Interview angle