Skip to content
ML to LLMs
Labs / Transformer Lab

Self-attention

Step through Q, K, V, QKᵀ, scaling, the causal mask, softmax, and the weighted sum, with shapes.

Theory and worked examples: Self-attention

Self-attention lab

Toy embeddings (d_model = 4) and random projections (d_k = 2), fixed seeds. Real models learn these.

X=E[tokens]+PX = E[\text{tokens}] + P
X: token embeddings + positions[6, 4]
d1d2d3d4
1:the-0.890.72-0.781.54
2:cat1.130.200.191.73
3:sat-1.53-1.35-0.090.71
4:on-0.76-1.101.511.46
5:the-1.65-0.93-0.741.54
6:mat-1.022.53-0.901.71
W_Q (learned in real models)[4, 2]
k1k2
d10.04-0.54
d2-0.240.91
d30.450.19
d40.490.61

Tokens and embeddings

Each token's embedding is a vector. Q, K and V are three different linear views of it: what this token is looking for, what it offers, and what it passes along.