Quizzes
LLM inference
Covers the lesson LLM inference.
Quiz: LLM inference
Question 1 of 10
Calculationmedium
A synthetic model has 40 layers, 40 query heads, 8 KV heads, and . The KV cache is stored in BF16 (2 bytes). How many bytes of KV cache does one token of one sequence need?