Learning path
Next moduleModule 26: Reinforcement learning
Module 25
Computer vision and multimodal
Detect, segment, and encode images; learn visual features without labels; align images with text; and connect vision encoders to language models.
- 1. Object detection and YOLOExplain how a one-stage detector turns a feature map into boxes, compute IoU and run non-maximum suppression by hand, encode and decode YOLO box targets, and read precision, recall, AP@0.5, and mAP@0.5:0.95.45 min
- 2. Semantic segmentation and U-NetLabel every pixel with a class, explain why U-Net pairs an encoder and decoder with skip connections, combine cross-entropy with Dice loss for imbalanced masks, and report per-class IoU.40 min
- 3. Vision transformersTurn an image into a sequence of patch tokens, trace shapes through a ViT, count its parameters, explain how patch size sets cost, and choose between ViT, hierarchical transformers, and CNNs.35 min
- 4. Self-supervised visionLearn visual features without labels using contrastive learning (SimCLR), self-distillation (DINO), and masked reconstruction (MAE); compute the InfoNCE loss by hand; and evaluate features with linear probes.40 min
- 5. CLIP and contrastive image-text pretrainingTrain an image encoder and a text encoder into one shared space with the symmetric contrastive loss, compute the similarity matrix, softmaxes, and loss by hand, use the result for zero-shot classification and retrieval, and compare softmax CLIP with sigmoid SigLIP.45 min
- 6. Vision-language modelsExplain how a vision encoder, a projector, and a language model combine into a VLM, trace image tokens into the LLM's sequence, budget image tokens and KV-cache memory, describe the training stages, and evaluate and monitor visual hallucination.40 min
Convolutional networks taught us to classify a whole image. This module covers what production vision systems actually do: find and outline objects, encode images with transformers, learn from unlabelled pictures, put images and text in one shared space, and let a language model read images.
| Task | Output | Typical model | Metric |
|---|---|---|---|
| Detection | labelled boxes | YOLO family, DETR family | mAP@0.5:0.95 |
| Semantic segmentation | class per pixel | U-Net, SegFormer | per-class IoU, Dice |
| Representation learning | feature vector per image | ViT with SimCLR, DINO, MAE | linear probe, k-NN |
| Image-text matching | shared embedding space | CLIP, SigLIP | zero-shot accuracy, recall@k |
| Visual question answering | text | ViT + projector + LLM | grounded accuracy, hallucination rate |
| # | Lesson | You will be able to |
|---|---|---|
| 1 | Object detection and YOLO | decode a YOLO head, compute IoU, run NMS by hand, and read mAP |
| 2 | Semantic segmentation and U-Net | trace U-Net shapes, explain skip connections, combine cross-entropy with Dice, and report per-class IoU |
| 3 | Vision transformers | turn an image into patch tokens, count ViT parameters, and budget attention cost by patch size |
| 4 | Self-supervised vision | compute InfoNCE, explain collapse and how DINO and MAE avoid it, and evaluate with linear probes |
| 5 | CLIP and contrastive image-text pretraining | compute the similarity matrix, both softmaxes, and the symmetric loss; run zero-shot classification; compare CLIP with SigLIP |
| 6 | Vision-language models | trace image tokens into an LLM, budget tokens and KV-cache memory, and evaluate visual hallucination |
Interactive lab: Contrastive image-text matching