Skip to content
ML to LLMs
Labs / LLM and RAG Lab

KV cache and inference memory

Size the weights and KV cache for a model, context, and batch; see whether it fits and how fast decoding can go.

Theory and worked examples: LLM inference

KV cache and inference memory

Synthetic model shapes and accelerator specs (illustrative, not specific products). GB = 10⁹ bytes; activations and overhead not included.

Weights plus KV cache versus context length, with accelerator capacity050100020406080context length T (thousand tokens)memory (GB)
weightsweights + KV cache, batch 16capacity (80 GB)current context

KV cache = 2 · 32 layers · 8 KV heads · 128 dims · 2 bytes · 8,192 tokens · 16 sequences = 17.18 GB. It fits with 46.8 GB to spare (minus activations and overhead), and the KV cache is 52% of the memory used.

Each decode step reads every weight and every cached key and value once, so speed is capped at bandwidth ÷ bytes read: about 90 tokens/s for each user and 1,447 tokens/s in total. With full multi-head attention the same KV cache would be 68.7 GB (4× larger).