Self-supervised vision

1. Intuition

In self-supervised learning the data provides its own targets through a pretext task. The pretext task does not matter in itself. It is useful only if solving it forces the network to learn features that transfer, such as shapes, parts, and object identity.

Three families have scaled well.

1. Contrastive (SimCLR, MoCo). Take a batch of $N$ images. Make two random views of each: random crop, flip, colour jitter, blur. Encode all $2N$ views. For each view, its twin is the positive; the other $2N - 2$ views are negatives. Train so each view is closer to its twin than to any negative. To succeed, the encoder must ignore what the augmentations change (colour, crop position) and keep what they preserve (it's a dog). More negatives make the task harder and the signal sharper, so SimCLR used batches of 4,096 or more. MoCo keeps a queue of recent embeddings from a slowly updated momentum encoder, which decouples the number of negatives from batch size.

2. Self-distillation (BYOL, DINO). No negatives at all. A student network sees one view and must predict the output of a teacher network that sees another view. The teacher is not trained by gradients: its weights are an exponential moving average (EMA) of the student's. The obvious cheat is collapse: output the same vector for every image and the student matches the teacher perfectly. DINO prevents this with two tricks on the teacher's output: centring (subtract a running mean, so no single dimension can win for every image) and sharpening (a low softmax temperature, so the output can't drift to uniform). Centring alone pushes toward uniform; sharpening alone pushes toward one dimension. Together they balance.

3. Masked image modelling (MAE, BEiT). Cut the image into patches as in a ViT, hide most of them, and reconstruct the hidden ones. MAE hides 75 percent. Images are highly redundant: with only 15 percent hidden, the network could fill each gap by blurring its neighbours, without understanding anything. At 75 percent it has to know what objects look like. MAE's encoder sees only the visible 25 percent, which makes pretraining cheap; a small decoder fills in the rest and is discarded after pretraining.

How do we know the features are good? Freeze the encoder and train only a linear classifier on top: a linear probe. If a straight line separates the classes in feature space, the encoder already did the hard work. A k-nearest-neighbour classifier on frozen features is an even simpler check. Fine-tuning the whole network gives higher numbers but mixes in the effect of further training.

Note

The families have different strengths. Contrastive and self-distillation methods produce features that work well frozen (good linear probes, retrieval). MAE features tend to need fine-tuning to shine but scale cheaply and suit dense tasks. DINOv2 combines self-distillation with a masked-patch objective (from iBOT) on a large curated dataset, which is why it is a common frozen backbone for retrieval, depth, and segmentation.

2. Visualization

Lower temperature rewards well-aligned views and punishes unrelated ones much harder. MAE keeps 49 of 196 patches visible.

Try it (predict first, then check with the script in section 4):

1. Random views at $\tau = 0.1$ score 4.39, above the chance level of 3.43. Predict why "random" can be worse than "uniform guessing".
2. Double the batch from 16 to 32 pairs. What happens to the chance-level loss, and what does that say about comparing loss values across batch sizes?
3. In the DINO demo, set the sharpening temperature to 0.5 instead of 0.04. Predict the entropy compared with 0.101 (sharp) and 2.768 (no sharpening).
4. Change the MAE ratio to 0.5. How much more attention compute does the encoder spend?

3. The math

Symbols

InfoNCE (NT-Xent), as in SimCLR

It is cross-entropy where the "classes" are the other $2N - 1$ views and the correct class is the twin. If the softmax were uniform, every view would score $\log(2N - 1)$: that is the chance level.

DINO teacher update and output

The student minimises the cross-entropy between $P_t$ on one view and its own $P_s$ on another. $\tau_t$ is small (0.04 to 0.07), $\tau_s$ larger (0.1).

MAE

The mean squared error over masked patches $M$ only; $|M| = rN$. The encoder processes $(1 - r)N$ patches, so its attention cost falls by about $1/(1-r)^2$: 16 times for $r = 0.75$.

Worked example: InfoNCE with N = 2

Two images, so four views, all already unit length, $\tau = 0.5$:

• image A: $a_1 = (1, 0)$, $a_2 = (0.8, 0.6)$
• image B: $b_1 = (0, 1)$, $b_2 = (0.6, 0.8)$

Cosines (dot products): $a_1 \cdot b_1 = 0$, $a_1 \cdot a_2 = 0.8$, $a_1 \cdot b_2 = 0.6$, $b_1 \cdot a_2 = 0.6$, $b_1 \cdot b_2 = 0.8$, $a_2 \cdot b_2 = 0.48 + 0.48 = 0.96$.

View $a_1$ (twin $a_2$). Candidates: $b_1, a_2, b_2$ with logits $0/0.5 = 0$, $0.8/0.5 = 1.6$, $0.6/0.5 = 1.2$.

1. Exponentials: $e^0 = 1$, $e^{1.6} = 4.953$, $e^{1.2} = 3.320$. Sum $= 9.273$.
2. Probability on the twin: $4.953 / 9.273 = 0.534$.
3. $\ell_{a_1} = -\ln 0.534 = 0.627$.

View $a_2$ (twin $a_1$). Candidates: $a_1, b_1, b_2$ with logits $1.6$, $1.2$, $0.96/0.5 = 1.92$.

1. Exponentials: $4.953$, $3.320$, $e^{1.92} = 6.821$. Sum $= 15.094$.
2. Probability on the twin: $4.953 / 15.094 = 0.328$.
3. $\ell_{a_2} = -\ln 0.328 = 1.114$.

By symmetry $\ell_{b_1} = 0.627$ and $\ell_{b_2} = 1.114$. The loss is $(0.627 + 1.114 + 0.627 + 1.114)/4 = 0.871$, below the chance level $\ln 3 = 1.099$.

View $a_2$ is the hard one: its closest neighbour is $b_2$ (cosine 0.96), a negative. The gradient pushes $a_2$ and $b_2$ apart and pulls $a_2$ toward $a_1$. Hard negatives like this carry most of the learning signal, which is why more negatives help.

4. Implementation

The full script is code/25-computer-vision/ssl_scratch.py (NumPy only). It implements InfoNCE, reproduces the worked example, sweeps temperature and batch size, builds an MAE mask, and shows DINO's collapse and its fix.

Build it from scratch

Step 1: InfoNCE. Stack both views into one $[2N, D]$ matrix, compute all pairwise cosines at once, mask the diagonal (a view is not its own candidate), and run cross-entropy where the target for row $i$ is its twin $N$ rows away.

def info_nce(z1, z2, tau=0.1):
    """z1, z2: [N, D] unit vectors; row i of z1 and row i of z2 are two views of image i."""
    n = len(z1)
    z = np.concatenate([z1, z2])                      # [2N, D]
    sim = z @ z.T / tau                               # [2N, 2N] cosine / temperature
    np.fill_diagonal(sim, -np.inf)                    # a view is never its own candidate
    target = np.concatenate([np.arange(n, 2 * n), np.arange(n)])   # view i's twin sits N rows away
    return float(-log_softmax(sim)[np.arange(2 * n), target].mean())

Predict: if you forgot fill_diagonal, each row's largest logit would be its self-similarity $1/\tau$. With identical views, what would the loss approach as $\tau \to 0$?

Step 2: MAE masking. A random permutation, keep the first 25 percent. Sorting keeps the visible patches in grid order, which makes adding position embeddings straightforward.

def random_mask(num_patches, ratio=0.75, seed=0):
    perm = np.random.default_rng(seed).permutation(num_patches)
    keep = int(num_patches * (1 - ratio))
    return np.sort(perm[:keep]), np.sort(perm[keep:])

Predict: MAE adds position embeddings to the visible patches before dropping the masked ones. Why must positions be attached before the encoder, rather than after?

Step 3: DINO's teacher output. Subtract the centre, divide by a temperature, softmax. The demo gives every sample a shared offset in one dimension to mimic the start of collapse, then measures how many distinct "winning" dimensions remain and how peaked each output is.

def teacher_probs(logits, centre, temp):
    return np.exp(log_softmax((logits - centre) / temp))

def collapse_report(p):
    entropy = -(p * np.log(p + 1e-12)).sum(-1).mean()
    return len(set(p.argmax(-1))), entropy

Predict: with centring but a very high temperature (10), every output is close to uniform. Why is a uniform target also a form of collapse for the student?

Use the library

Pretrained self-supervised encoders are a download away. Use them frozen for retrieval and probes, or fine-tune.

import torch
from transformers import AutoImageProcessor, AutoModel       # pip install transformers torch pillow

proc = AutoImageProcessor.from_pretrained("facebook/dinov2-base")
model = AutoModel.from_pretrained("facebook/dinov2-base").eval()
with torch.no_grad():
    out = model(**proc(images=pil_images, return_tensors="pt"))
feats = torch.nn.functional.normalize(out.last_hidden_state[:, 0], dim=-1)   # [B, 768] CLS features

# linear probe: freeze feats, fit any linear classifier
from sklearn.linear_model import LogisticRegression
probe = LogisticRegression(max_iter=2000).fit(train_feats, train_labels)

What the libraries add: the augmentation pipelines (multi-crop for DINO), distributed negatives or queues, EMA teachers, the masking and decoder for MAE, and, most importantly, weights pretrained on far more images than you could process yourself. timm hosts MAE and DINO checkpoints; lightly packages SimCLR, MoCo, BYOL, and DINO training loops.

Run it

uv run code/25-computer-vision/ssl_scratch.py

Key lines of output (synthetic embeddings):

InfoNCE = 0.8707
identical views: 0.0073
random views:    4.3862   (chance level log(2N - 1) = log 31 = 3.4340)
tau 0.05: aligned 0.9355   random 7.9128
tau 1.0 : aligned 3.0036   random 3.4590
batch N = 4096: 8190 negatives, chance-level loss log(2N - 1) = 9.011
196 patches -> encoder sees 49, decoder reconstructs 147; first visible: [0, 5, 6, 10, 13, 35]
encoder self-attention scores: full 38,809 vs visible-only 2,500 (15.5x fewer)
no centring, sharp (temp 0.04)       distinct argmax =  1 / 16, mean entropy = 0.000 (uniform = 2.773)
centring, no sharpening (temp 10)    distinct argmax = 16 / 16, mean entropy = 2.768 (uniform = 2.773)
centring + sharpening (temp 0.04)    distinct argmax = 16 / 16, mean entropy = 0.101 (uniform = 2.773)

Random views score above chance at low temperature because their logits are spread out: the twin is often far from the top, and a confident wrong softmax costs more than a uniform one.

5. Engineering

When to pretrain yourself. Rarely from scratch. Start from a public self-supervised checkpoint. Continue self-supervised pretraining on your own unlabelled images when your domain looks unlike web photos (medical scans, satellite tiles, microscopy, industrial cameras), then fine-tune or probe with your labels.

Augmentations define the invariances. Contrastive and self-distillation methods learn to ignore whatever the augmentations change. Colour jitter teaches colour invariance, which is wrong if colour is the label (ripe versus unripe fruit, some lesions). Choose augmentations per domain.

Compute. Contrastive methods need many negatives: large batches across GPUs or a memory queue. DINO needs several crops per image (two global, several local). MAE is the cheapest per step because the encoder sees a quarter of the patches; pretraining throughput is several times higher than encoding every patch.

Evaluation. Report a linear probe and k-NN accuracy on frozen features, plus fine-tuned accuracy on the target task. Use a held-out labelled set that was never used to pick augmentations or checkpoints.

Watch out

Failure modes. Collapse (all embeddings identical: watch the per-dimension standard deviation of the embeddings, which falls toward zero); shortcut learning (views from the same image share a colour histogram or a JPEG artefact the network can match without understanding content); batch-norm information leaking between views in contrastive training (MoCo used shuffling BN to prevent it); false negatives (two different photos of the same product treated as negatives).

Common mistakes

• Comparing InfoNCE values across batch sizes. The chance level is $\log(2N - 1)$, so a bigger batch has a bigger loss for the same quality.
• Judging a self-supervised model by its pretext loss instead of a downstream probe.
• Copying ImageNet augmentation settings to a domain where colour or orientation is meaningful.
• Using MAE features frozen and concluding they are weak; they usually need fine-tuning.
• Forgetting to normalise embeddings before the dot product, so the "cosine" is unbounded.

6. Knowledge check

Practice exercise. A batch has $N = 128$ images. How many negatives does each view have in SimCLR? What InfoNCE loss would a model that outputs uniform probabilities get? If a model assigns probability 0.25 to the twin on average, roughly what is its loss?

Implementation challenge. Implement the BYOL / DINO EMA teacher update for a weight dictionary and measure how fast the teacher follows the student: set student weights to all ones, teacher to zeros, and apply 100 updates with $m = 0.99$ and then $m = 0.996$. What fraction of the gap is closed?

def ema(teacher, student, m):
    return {k: m * teacher[k] + (1 - m) * student[k] for k in teacher}

t, s = {"w": np.zeros(3)}, {"w": np.ones(3)}
for _ in range(100):
    t = ema(t, s, 0.99)
print(t["w"][0])   # 1 - 0.99**100 = 0.634

Summary

• Self-supervised learning makes its own targets: match two views (contrastive), match an EMA teacher (self-distillation), or reconstruct hidden patches (masked image modelling).
• InfoNCE is cross-entropy over $2N - 1$ candidates; temperature sets how hard it punishes near misses, and more negatives give a sharper signal.
• Collapse is the central risk; negatives, centring plus sharpening, or a reconstruction target prevent it.
• MAE masks 75 percent because images are redundant, and it encodes only visible patches, which makes it cheap.
• Judge features by linear probe, k-NN, and fine-tuning on held-out labels, never by the pretext loss.

Next: CLIP and contrastive image-text pretraining

Related: Vision transformers · Learning paradigms · Word embeddings · Autoencoders, diffusion, and transfer

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Self-Supervised Vision — SimCLR, DINO, MAE.

Interview angle