Learning path
Next moduleModule 15: Large language models
Module 14
Transformers
Self-attention, multi-head attention, positional encoding, and the full transformer block.
- 1. Self-attentionCompute scaled dot-product attention by hand, track the shape of Q, K, V and every intermediate matrix, and explain scaling, masking, and multiple heads.45 min
- 2. The transformer architectureTrace a token through embeddings, positional information, attention, residual connections, layer normalization, and the feed-forward network; count a block's parameters; and compare encoder, decoder, and encoder-decoder models.40 min
- 3. BERT and masked language modelingPretrain an encoder by hiding tokens and predicting them from both sides, implement the 80/10/10 masking rule and the loss mask, and decide when an encoder beats a decoder.35 min
- 4. Attention variantsDraw causal, sliding-window, strided, and global-token attention masks, count what each query can see, and size the compute, KV cache, and receptive field each one buys.40 min
Transformers let every token gather information from every other relevant token through attention, while processing all positions in parallel during training. That combination made them the architecture behind modern language, vision, and multimodal models.
| # | Lesson | You will be able to |
|---|---|---|
| 1 | Self-attention | compute scaled dot-product attention by hand with every shape |
| 2 | The transformer architecture | trace a token through embeddings, attention, residuals, normalization, and the feed-forward network, and compare encoder, decoder, and encoder-decoder models |
| 3 | BERT and masked language modeling | implement the 80/10/10 masking rule and MLM loss, and choose an encoder or a decoder for a task |
| 4 | Attention variants | draw causal, sliding-window, strided, and global-token masks and size their compute, KV cache, and receptive field |
Interactive labs: Self-attention · Attention masks