Labs / Transformer Lab
Contrastive image-text matching
Build the image-text similarity matrix, apply temperature and softmax, and see how the contrastive loss pulls matching pairs together.
Theory and worked examples: CLIP and contrastive image-text pretraining
Contrastive image-text matching
SYNTHETIC 5-D embeddings stand in for an image encoder and a text encoder. Train steps run gradient descent on the symmetric loss at τ = 0.1.
each row sums to 1: image → text. Diagonal = matching pairs (outlined); dot = best match in that row.
“a photo of a cat”
“a photo of a dog”
“a red car on a street”
“a slice of pizza”
“a tree in a park”
0.41
0.59
0.00
0.00
0.00
0.22
0.78
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.01
0.00
0.99
Zero-shot: rank the captions for 🐱 (click an image to change)
“a photo of a dog”
0.591
“a photo of a cat”
0.408
“a tree in a park”
0.000
“a red car on a street”
0.000
“a slice of pizza”
0.000
Teal = the correct caption. Probabilities are this image's row softmax at τ = 0.10; training uses τ = 0.1.