Labs / LLM and RAG Lab
Static vs continuous batching
Simulate requests of different lengths on a GPU and compare throughput and latency under each batching policy.
Theory and worked examples: Serving LLMs
Static vs continuous batching
30 synthetic requests, mean output 24 tokens. One step = one decode iteration; each busy slot emits one token per step.
Each bar is one request occupying a slot from admission to its last token. Blank space is an idle slot. Hover a bar for its arrival time.