Skip to content
ML to LLMs
Labs / Transformer Lab

Attention masks and variants

Compare full, causal, sliding-window, and sparse attention masks; see which tokens each query can reach and how cost grows with length.

Theory and worked examples: Attention variants

Attention mask patterns

Rows are queries, columns are keys. A coloured cell means that query may attend to that key. Click a row to pick a query.

key position →

what query 12 reads directly allowed for other queries
Causal attention, as in a GPT-style decoder: query 12 sees 13 keys, never the future. Still about n²/2 work.