Skip to content
ML to LLMs
Quizzes

LLM inference

Covers the lesson LLM inference.

Quiz: LLM inference

Question 1 of 10
Calculationmedium

A synthetic model has 40 layers, 40 query heads, 8 KV heads, and dhead=128d_{\text{head}} = 128. The KV cache is stored in BF16 (2 bytes). How many bytes of KV cache does one token of one sequence need?