Convolutional networks

Quick read: the intuition, interactive visualization, summary, and interview questions. The math, implementation, engineering, and knowledge-check sections are hidden.

Mental model. A convolution slides a small pattern detector (a kernel) across an image and records how strongly each location matches. The same detector is reused everywhere, so a CNN learns "what an edge looks like" once and finds it anywhere.

You will learn to

Compute a convolution output cell by hand and a full feature map.
Compute output sizes from input size, kernel size, stride, and padding.
Apply max pooling and explain why it is used.
Count parameters in a convolutional layer and compare with a dense layer.
Explain the inductive biases (locality, weight sharing) that make CNNs data-efficient on images.

Why it matters. CNNs made modern computer vision work and remain strong, efficient backbones for images, audio spectrograms, and some sequence tasks. The ideas of local receptive fields and weight sharing reappear across deep learning.

1. Intuition

A dense layer connects every pixel to every unit: a 224×224 color image has 150,528 inputs, so each hidden unit needs 150,528 weights, and the layer ignores that nearby pixels are related.

A convolutional layer instead uses small kernels, say 3×3. Each kernel looks at one 3×3 patch at a time, computes a weighted sum, then slides over. The result is a feature map showing where the kernel's pattern appears. A layer has many kernels (channels), each learning a different pattern. Early layers learn edges and colors; deeper layers combine them into textures, parts, and objects.

Stride is how far the kernel moves each step (stride 2 halves the resolution). Padding adds zeros around the border so the output can keep the input's size. Pooling downsamples by taking the maximum (or average) of small windows, keeping the strongest responses and adding a little tolerance to small shifts.

2. Visualization

Convolution and pooling

Click any output cell (or use the slider) to see the 3×3 window that produced it.

input image[7, 7]

kernel[3, 3]

feature map[5, 5]

2×2 max pool[2, 2]

Kernel, feature map and pool: teal is positive, orange is negative, stronger colour means a larger magnitude. The purple outline is the current window and the output cell it produces.

Try it (predict first, then check):

Before clicking, predict the feature-map value where the window sits entirely on the bright region. Then click that cell.
Switch the image to the horizontal edge but keep the vertical-edge kernel. Predict the output, then check.
Set stride 2, then padding 1. Use the size formula to predict each output size before you look.

3. The math

Symbols

Symbol	Meaning
$X$	input of size $H \times W$ (one channel)
$K$	kernel of size $k \times k$
$s$ , $p$	stride and padding
$C_{\text{in}}, C_{\text{out}}$	input and output channels
$Y$	output feature map

Convolution (as deep-learning libraries compute it: cross-correlation)

Y_{ij} = \sum_{u=0}^{k-1}\sum_{v=0}^{k-1} X_{\,i s + u,\; j s + v}\; K_{uv} \;+\; b

Output size

H_{\text{out}} = \left\lfloor \frac{H - k + 2p}{s} \right\rfloor + 1

Parameters

A conv layer has $C_{\text{out}} \times (C_{\text{in}} \times k \times k + 1)$ parameters, independent of image size.

Worked example 1: one output cell

Input (5×5): columns 1 to 3 are 0, columns 4 to 5 are 9. Kernel rows are all $[-1, 0, 1]$ .

Top-left cell: the window is all zeros, so $Y_{00} = 0$ .
Top-middle cell: the window covers columns 2 to 4, each row $[0, 0, 9]$ . Per row: $(-1)(0) + (0)(0) + (1)(9) = 9$ . Three rows: $Y_{01} = 27$ .
Top-right cell: columns 3 to 5, each row $[0, 9, 9]$ : $0 + 0 + 9 = 9$ per row, so 27.

Full feature map: every row is $[0, 27, 27]$ .

Size check: $(5 - 3 + 0)/1 + 1 = 3$ . With padding 1: $(5 - 3 + 2)/1 + 1 = 5$ . With stride 2 and no padding: $\lfloor 2/2 \rfloor + 1 = 2$ .

Worked example 2: max pooling

A 4×4 map with 2×2 windows and stride 2:

\begin{bmatrix}1&3&2&1\\4&6&5&0\\1&2&9&8\\3&1&4&7\end{bmatrix} \;\to\; \begin{bmatrix}\max(1,3,4,6) & \max(2,1,5,0)\\ \max(1,2,3,1) & \max(9,8,4,7)\end{bmatrix} = \begin{bmatrix}6&5\\3&9\end{bmatrix}

Worked example 3: parameter count

A layer with 64 input channels, 128 output channels, and 3×3 kernels: $128 \times (64 \times 9 + 1) = 128 \times 577 = 73{,}856$ parameters. A dense layer from a $32 \times 32 \times 64$ input to an output of the same size would need over 4 billion weights.

4. Implementation

The full script is code/13-deep-architectures/conv_scratch.py (NumPy only). It builds a real multi-channel convolution, rewrites it as one matrix multiply the way fast libraries do, and adds pooling, receptive fields, and a residual connection.

Build it from scratch

Step 1: the output-size formula. Count how many window positions fit. Every shape in a CNN comes from this one line.

python

def out_size(n, k, pad=0, stride=1):
    return (n + 2 * pad - k) // stride + 1

Predict: a 32×32 image, 5×5 kernel, no padding, stride 1. What size is the output? What padding keeps it at 32?

Step 2: multi-channel convolution. The input is [C_in, H, W] and the weights are [C_out, C_in, k, k]. Each filter spans all input channels, so one output pixel sums C_in × k × k products. Each of the C_out filters produces one feature map.

python

def conv2d(x, w, b=None, stride=1, pad=0):
    c_out, _, k, _ = w.shape
    x = np.pad(x, ((0, 0), (pad, pad), (pad, pad)))
    oh, ow = out_size(x.shape[1], k, 0, stride), out_size(x.shape[2], k, 0, stride)
    out = np.zeros((c_out, oh, ow))
    for o in range(c_out):                                  # one feature map per filter
        for i in range(oh):
            for j in range(ow):
                patch = x[:, i * stride:i * stride + k, j * stride:j * stride + k]
                out[o, i, j] = np.sum(patch * w[o])         # sums over channels too
    return out if b is None else out + b[:, None, None]

Predict: on worked example 1's step image, what does this return? (The script prints it.)

Step 3: im2col. Three nested Python loops are slow. Copy every window into one column of a matrix, flatten each filter into a row, and the whole convolution becomes a single matrix multiply that BLAS or a GPU runs fast. The cost is memory: each input pixel is copied up to $k^2$ times.

python

def conv2d_im2col(x, w, stride=1, pad=0):
    c_out, c_in, k, _ = w.shape
    x = np.pad(x, ((0, 0), (pad, pad), (pad, pad)))
    oh, ow = out_size(x.shape[1], k, 0, stride), out_size(x.shape[2], k, 0, stride)
    cols = np.stack([x[:, i * stride:i * stride + k, j * stride:j * stride + k].ravel()
                     for i in range(oh) for j in range(ow)], axis=1)   # [C_in*k*k, oh*ow]
    return (w.reshape(c_out, -1) @ cols).reshape(c_out, oh, ow)

Predict: for a [3, 8, 8] input, a 3×3 kernel, and padding 1, what shape is cols?

Step 4: receptive field. Each layer widens what one output pixel can "see" by k - 1 steps of its input. Strides and pooling make later steps cover more pixels. This is why stacks of small kernels replaced large ones: three 3×3 layers see 7×7 with fewer weights and two extra nonlinearities.

python

def receptive_field(layers):                                # layers: [(kernel, stride), ...]
    rf, jump = 1, 1
    for k, s in layers:
        rf += (k - 1) * jump                                # each layer widens by (k-1) input steps
        jump *= s                                           # stride spreads later layers out
    return rf

Predict: what is the receptive field of conv3 → pool2 → conv3 → pool2 → conv3?

Step 5: a residual block. ResNet's idea is $y = x + F(x)$ . If $F$ 's weights are zero, the block passes its input straight through. A deep stack therefore starts close to the identity and gradients have a direct path back, which is what made 50 to 150 layer networks trainable.

python

    y = relu(x + conv2d(relu(conv2d(x, zero, pad=1)), zero, pad=1))
    print("output equals input:", bool(np.allclose(y, x)))

Predict: why does this need the input x to be non-negative (it went through a ReLU) for the output to equal it exactly?

Use the library

nn.Conv2d is step 2 with learned weights; under the hood cuDNN picks among im2col-style GEMM, Winograd, and FFT algorithms per layer shape. Shapes follow the formula from step 1.

python

import torch
from torch import nn

cnn = nn.Sequential(
    nn.Conv2d(3, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),   # [B,3,32,32] -> [B,32,16,16]
    nn.Conv2d(32, 64, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # -> [B,64,8,8]
    nn.Flatten(), nn.Linear(64 * 8 * 8, 10),                                  # -> [B,10] logits
)
print(cnn(torch.randn(4, 3, 32, 32)).shape)   # torch.Size([4, 10])

Run it

bash

uv run code/13-deep-architectures/conv_scratch.py

Key lines of the output:

text

[[ 0. 27. 27.]
 [ 0. 27. 27.]
 [ 0. 27. 27.]]
sizes: no pad 3  pad 1 5  stride 2 2
output shape: (4, 8, 8)  max difference: 5.3e-15
conv: 73856  dense 32x32x64 -> 32x32x64: 4,294,967,296
three 3x3 stride-1 convs: 7  (same view as one 7x7)
params, C=64 channels: three 3x3 = 110592  one 7x7 = 200704
output equals input: True

The feature map, sizes, pooling, and parameter count match the three worked examples. The loop and im2col agree to floating-point rounding. The figure script is code/13-deep-architectures/architectures.py.

5. Engineering

Use pretrained models. For most image tasks, start from a pretrained backbone (ResNet, EfficientNet, ConvNeXt, or a vision transformer) and fine-tune; training from scratch needs far more data (see transfer learning).

Augmentation (random crops, flips, color jitter) is essential for CNNs on small datasets, as long as it preserves the label. Mixup goes further: it blends two images and their one-hot labels with the same weight (say 0.7 and 0.3) and trains on the soft target, which smooths decision boundaries and reduces overconfidence.

Architecture pieces. Batch normalization after convolutions; residual connections for depth; global average pooling instead of large dense heads; 1×1 convolutions to mix channels cheaply.

CNNs versus vision transformers. CNNs' built-in locality makes them data-efficient and fast; vision transformers scale well with very large data. Hybrids are common.

Watch out
Failure modes. Off-by-one size errors after stride and padding; augmentations that change labels; shortcuts (the model detects a watermark or background instead of the object); distribution shift between training photos and production cameras.

Common mistakes

Forgetting that channels are a dimension: inputs are $[B, C, H, W]$ in PyTorch.
Computing the flattened size by hand wrong; run a dummy tensor through and print shapes.
Training a large CNN from scratch on a few thousand images.
Applying random augmentation at evaluation time, or computing the normalization mean and standard deviation on train plus test. Augment only the training split, and compute statistics from it alone (or use the pretrained model's own values).

6. Knowledge check

Quiz: Convolutional networks

Question 1 of 6

Calculationmedium

A 2×2 kernel $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ is applied to the patch $\begin{bmatrix}5 & 2\\3 & 1\end{bmatrix}$ (no bias). What is the output value?

Practice exercise. A 32×32 input passes through a 5×5 convolution with stride 1 and no padding, then 2×2 max pooling with stride 2. What is the output size?

Solution

Convolution: $(32 - 5)/1 + 1 = 28$ , so 28×28. Pooling: $28/2 = 14$ , so 14×14.

Implementation challenge. Apply the vertical-edge kernel and its transpose (a horizontal-edge kernel) to a real grayscale photo with your conv2d, display both feature maps, and compare with scipy.signal.correlate2d.

Summary

A convolution slides a shared kernel over the input; each output cell is a patch-kernel dot product plus bias.
Output size $= \lfloor (H - k + 2p)/s \rfloor + 1$ ; parameters don't depend on image size.
Pooling downsamples and adds small-shift tolerance.
Locality and weight sharing make CNNs efficient for images; in practice, fine-tune pretrained backbones.

Next: Recurrent networks, LSTMs, and GRUs

Go deeper, code-first (AI Engineering from Scratch): Image Fundamentals — Pixels, Channels, Color Spaces

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Convolutions from Scratch, CNNs — LeNet to ResNet, Image Classification.

Interview angle

Why do CNNs need far less data than an MLP to learn from images?

Because they build in two assumptions that are true of images: locality and translation equivariance. Locality means each output depends on a small $k \times k$ patch, since nearby pixels are related. Weight sharing means the same kernel is applied at every location, so a pattern learned in one corner is recognized everywhere, and shifting the input shifts the feature map. Pooling adds some tolerance to small shifts. These biases shrink the hypothesis space dramatically. A 3×3 conv from 64 to 128 channels has $128 \times (64 \times 9 + 1) = 73{,}856$ parameters regardless of image size; a dense layer from a $32 \times 32 \times 64$ input to an output of the same size needs over 4 billion. Fewer free parameters, each trained on every spatial location, means less data to fit them. Vision transformers have weaker biases and catch up only with much larger datasets or pretraining.

An input of 3×224×224 goes through a 7×7 convolution with 64 filters, stride 2, padding 3, then 3×3 max pooling with stride 2 and padding 1. What are the output shapes and parameter count?

Use $H_{\text{out}} = \lfloor (H - k + 2p)/s \rfloor + 1$ . Convolution: $\lfloor (224 - 7 + 6)/2 \rfloor + 1 = \lfloor 223/2 \rfloor + 1 = 111 + 1 = 112$ , so the output is $[64, 112, 112]$ . Pooling: $\lfloor (112 - 3 + 2)/2 \rfloor + 1 = \lfloor 111/2 \rfloor + 1 = 55 + 1 = 56$ , giving $[64, 56, 56]$ . Parameters: $C_{\text{out}} \times (C_{\text{in}} \times k \times k + 1) = 64 \times (3 \times 49 + 1) = 64 \times 148 = 9{,}472$ . Pooling has none. This is the standard ResNet stem: it cuts spatial size 4x in each dimension before the expensive layers, so later convolutions run on 16x fewer positions. In code, don't trust hand arithmetic for flattened sizes; push a dummy tensor of shape $[1, 3, 224, 224]$ through and print the shapes.

Your image classifier scores 98% on validation, but fails on photos from the production camera. What do you check?

Assume distribution shift or a shortcut until proven otherwise. First, preprocessing parity: same resize and crop, same normalization mean and standard deviation as training, RGB versus BGR channel order, same resolution, and same JPEG compression. A mismatch here is the cheapest bug to find. Second, shortcut learning: did the model learn a background, watermark, or camera artifact correlated with the label in the training set? Saliency maps such as Grad-CAM, or testing on images with backgrounds swapped, reveal this. Third, real shift: lighting, angle, sensor noise, or class balance differ in production. Collect a few hundred labeled production images as a new evaluation slice, measure the gap, and look at the confusion matrix. Fixes include augmentation that mimics the camera (blur, color jitter, noise), fine-tuning on production samples, and keeping that slice as a permanent regression test.

You have 5,000 labeled images for a new classification task. CNN or vision transformer?

Either, as long as it is pretrained; training anything from scratch on 5,000 images is the real mistake. A pretrained CNN (ResNet, EfficientNet, ConvNeXt) is a strong, cheap default: its locality bias makes it data-efficient, it handles high resolution efficiently, and it is fast on edge hardware. A pretrained vision transformer can match or beat it when pretrained on very large data, and it captures global context from the first layer, but plain self-attention cost grows quadratically with the number of patches. With 5,000 images, I'd fine-tune both a ConvNeXt-class CNN and a comparably sized pretrained ViT using the same augmentation and schedule, and pick by validation accuracy at the latency budget. Start with a frozen backbone and linear head as the baseline, then fine-tune the top layers or everything with a small learning rate. Measure latency on the target hardware, not in FLOPs.