Serving LLMs

1. Intuition

A request's life on a server: it waits in a queue, gets admitted, runs prefill on its prompt (producing the first token), then joins the running batch for decode steps until it emits an end token or hits its length limit. The previous lesson showed that one decode step costs about the same whether the batch holds 1 sequence or 32, so the goal is to keep the batch full of useful work.

Static batching collects up to $K$ requests, runs them together, and admits nobody new until all of them finish. Output lengths vary a lot (a "yes" next to a 900-token essay), so most slots go idle while the batch waits for its longest member, and new arrivals wait too.

Continuous batching (also called in-flight or iteration-level batching) makes the admission decision every decode step. When a sequence finishes, its slot is refilled on the next step. Slots go idle only when nobody is waiting.

Paged attention stores each sequence's KV cache in fixed-size blocks (for example 16 tokens) allocated on demand, with a per-sequence block table mapping logical positions to physical blocks, like virtual memory pages. Without it, servers reserved a contiguous region for each request's maximum length, and most of that memory sat empty. Paging also lets sequences share blocks.

Prefix caching reuses those shared blocks: if many requests start with the same system prompt, few-shot examples, or document, the KV for that prefix is computed once and reused. Hits skip prefill for the shared part. API providers expose the same idea as discounted "cached input tokens".

Chunked prefill splits a long prompt's prefill into chunks (say 512 tokens) and interleaves them with other sequences' decode steps. Without it, one 30k-token prompt stalls every user's next token while it prefills. Prefill-decode disaggregation goes further: separate GPU pools run prefill and decode, and the KV cache is transferred between them, so each pool can be sized and tuned for its own bottleneck.

When one GPU isn't enough:

Tensor parallelism also adds bandwidth: two GPUs read half the weights each, so per-token latency can drop, minus communication time.

Under load. When KV memory runs out, the scheduler must stop admitting, or preempt a running sequence: drop its cache and recompute it later, or swap it to CPU memory over PCIe. Long queues show up first as rising TTFT; oversized batches show up as rising TPOT.

2. Visualization

Try it (predict first, then check):

1. Load the Lesson example. Before switching policies, predict the finish step of each request under static batching. Then check your answers against the worked example below.
2. Choose Same lengths (σ = 0). Predict whether continuous batching still wins, and by how much compared with σ = 0.8.
3. Choose Near capacity and then raise the arrival rate past the capacity hint. Predict what happens to p95 latency under each policy.
4. Hold the arrival rate fixed and double the slots from 4 to 8. Which metric improves most: throughput, mean latency, or idle %?

3. The math

Symbols

The simulator's rules: one time step is one decode iteration, every occupied slot emits one token per step, and a request admitted at step $s$ finishes at $f = s + n$.

where $\ell_{(j)}$ is the $j$-th smallest latency (nearest-rank percentile).

Worked example: static versus continuous

$K = 2$ slots; four requests all arrive at step 0 with $n = [2, 8, 3, 3]$. Total tokens $2 + 8 + 3 + 3 = 16$.

Static. Batch 1 is requests 0 and 1, starting at 0. Request 0 finishes at $0 + 2 = 2$; request 1 at $0 + 8 = 8$. Slot 0 sits idle from 2 to 8 because the batch isn't done. Batch 2 starts at 8: request 2 finishes at $8 + 3 = 11$, request 3 at $8 + 3 = 11$.

• Finish steps $[2, 8, 11, 11]$; latencies equal finish steps since all arrived at 0.
• $M = 11$. Throughput $= 16 / 11 = 1.4545$ tokens/step.
• Slot capacity $K \cdot M = 2 \cdot 11 = 22$ slot-steps; idle $= 1 - 16/22 = 6/22 = 27.3\%$.
• Mean latency $= (2 + 8 + 11 + 11)/4 = 32/4 = 8.0$ steps. p95: $\lceil 0.95 \cdot 4 \rceil = 4$, the 4th smallest, $= 11$.

Continuous. Requests 0 and 1 start at 0. Request 0 finishes at 2; request 2 takes slot 0 at step 2 and finishes at $2 + 3 = 5$; request 3 takes slot 0 at 5 and finishes at $5 + 3 = 8$. Request 1 finishes at 8 in slot 1.

• Finish steps $[2, 8, 5, 8]$. $M = 8$. Throughput $= 16/8 = 2.0$ tokens/step, a $2.0 / 1.4545 = 1.375\times$ gain.
• Idle $= 1 - 16/(2 \cdot 8) = 1 - 16/16 = 0\%$.
• Mean latency $= (2 + 8 + 5 + 8)/4 = 23/4 = 5.75$ steps. p95 $= 8$.

Same hardware, same requests: 37.5% more throughput and 28% lower mean latency, just from the admission rule. The gain grows with the spread of output lengths.

Capacity and Little's law

A server with $K$ slots and mean output length $\bar{n}$ can finish at most $K / \bar{n}$ requests per step. In the figure, $4 / 24 = 0.167$. As $\lambda$ approaches capacity, queues grow without bound and latency explodes; static batching wastes slots, so its effective capacity is lower and it blows up earlier.

Little's law links the three quantities in any stable queue: average requests in the system $=$ arrival rate $\times$ average time in the system. With 2 requests/s and the previous lesson's 7.675 s end-to-end latency, the server holds $2 \times 7.675 = 15.35$ requests on average, so it needs a batch of about 16 and the KV memory for 16 sequences.

Paged KV memory

KV budget 40 GB, 131,072 bytes per token (the 8B-style GQA model in BF16), requests average 600 tokens of context, configured maximum 4,096.

• Contiguous reservation: each sequence reserves $4096 \cdot 131{,}072 = 536{,}870{,}912$ bytes. Sequences that fit: $\lfloor 40 \times 10^9 / 536{,}870{,}912 \rfloor = \lfloor 74.5 \rfloor = 74$. Useful fraction: $600 / 4096 = 14.6\%$.
• Paged, 16-token blocks: $\lceil 600/16 \rceil = 38$ blocks $= 608$ tokens $= 79{,}691{,}776$ bytes. Sequences that fit: $\lfloor 40 \times 10^9 / 79{,}691{,}776 \rfloor = \lfloor 501.9 \rfloor = 501$. Useful fraction: $600/608 = 98.7\%$.

About $6.8\times$ more concurrent sequences from the same memory. Waste is at most one partly filled block per sequence. (Real sequences grow while decoding, so the scheduler must also handle running out of blocks mid-generation, by preempting.)

Prefix caching

A 2,000-token shared system prompt plus 200 user tokens: prefill drops from 2,200 to 200 tokens on a cache hit, $1 - 200/2200 = 90.9\%$ less prefill compute, and TTFT falls by a similar fraction. The 2,000 tokens of KV are stored once and shared by every request that hits.

Cost per million tokens

Synthetic numbers: one GPU at $4.00 per hour sustaining 1,500 output tokens/s at your latency SLO.

• Tokens per hour: $1500 \times 3600 = 5{,}400{,}000$.
• Cost per million at 100% busy: $4.00 / 5.4 = 0.741$, so $0.741.
• Real traffic has peaks and valleys. At 60% average utilization: $0.741 / 0.6 = 1.23$, so $1.23; at 30%: $0.741 / 0.3 = 2.47$, so $2.47.
• One GPU for a month (730 hours) costs $4.00 \times 730 = 2{,}920$, so $2,920, and can produce at most $5.4\text{M} \times 730 = 3.942$ billion tokens. Against a synthetic API price of $1.00 per million output tokens, break-even is $2{,}920$ million tokens per month, which is $2{,}920 / 3{,}942 = 74\%$ utilization, before engineering and on-call time.

Utilization, not the hourly price, usually decides whether self-hosting pays.

4. Implementation

Build it from scratch

Continuous batching. The core of the simulator, with the same rules as the lab. Each slot is one sequence in the running batch; static admits new work only when every slot is free, continuous refills any free slot every step:

def simulate(requests, slots, policy):            # requests: list of (arrival, n_tokens)
    queue, nxt, t = [], 0, 0
    busy_until, finish = [0] * slots, [None] * len(requests)
    order = sorted(range(len(requests)), key=lambda i: requests[i][0])
    while None in finish:
        while nxt < len(order) and requests[order[nxt]][0] <= t:
            queue.append(order[nxt]); nxt += 1
        free = [s for s in range(slots) if busy_until[s] <= t]
        if policy == "continuous" or len(free) == slots:   # static waits for the whole batch
            for s in free[:len(queue)]:
                i = queue.pop(0)
                busy_until[s] = finish[i] = t + requests[i][1]
        t += 1
    return finish

Predict: with 4 slots and one 400-token request among three 50-token ones, how many steps does each policy leave three slots idle?

The rest of the build is code/18-llm-inference/serving_scratch.py (stdlib only, synthetic requests).

Paged KV memory. Instead of reserving a max-length slab per request, a pool hands out 16-token blocks as the sequence grows. Each request keeps a block table mapping its logical blocks to physical ones, like an operating system's page table. That is the core idea of vLLM's PagedAttention.

class BlockPool:
    def __init__(self, n_blocks):
        self.free = list(range(n_blocks))
        self.tables = {}  # request id -> list of physical block ids (its "page table")

    def grow(self, rid, n_tokens):
        """Make sure request rid owns enough blocks for n_tokens. False means: preempt someone."""
        table = self.tables.setdefault(rid, [])
        need = -(-n_tokens // BLOCK) - len(table)  # ceil division
        if need > len(self.free):
            return False
        table += [self.free.pop() for _ in range(need)]
        return True

    def release(self, rid):
        self.free += self.tables.pop(rid, [])

Predict: a request with 600 tokens of context holds how many blocks, and how many token slots are wasted? (38 blocks, 8 slots.)

Prefix caching. Two prompts can share KV blocks only if every earlier token matches too, because each key and value depends on the whole prefix. So hash each full block together with the hash of everything before it. A lookup walks the chain until the first miss.

def block_hashes(tokens):
    h, out = 0, []
    for i in range(0, len(tokens) - len(tokens) % BLOCK, BLOCK):
        h = hash((h, tuple(tokens[i : i + BLOCK])))  # chain: same hash only if the whole prefix matches
        out.append(h)
    return out

def prefill_with_cache(cache, tokens):
    """Return how many prompt tokens were served from cache, then store this prompt's blocks."""
    hit = 0
    for h in block_hashes(tokens):
        if h not in cache:
            break  # first miss ends the shared prefix
        hit += BLOCK
    cache.update(block_hashes(tokens))
    return hit

Predict: if one token near the start of the system prompt changes (say, today's date), what happens to the hit rate? Why do teams put dynamic content at the end of the prompt? SGLang's RadixAttention stores the same information as a radix tree over tokens, which also makes partial-block matches and LRU eviction of leaves natural.

Latency metrics and goodput. From each request's send time and token arrival times, compute TTFT, TPOT, and end-to-end latency. Goodput is the share of requests that met every SLO at once. That is the number users feel; throughput alone can rise while goodput falls.

def metrics(trace):
    """trace = (send time, list of token arrival times). TPOT excludes the first token."""
    t0, toks = trace
    ttft = toks[0] - t0
    tpot = (toks[-1] - toks[0]) / (len(toks) - 1)
    return ttft, tpot, toks[-1] - t0

def goodput(traces, ttft_slo, tpot_slo):
    ok = [m[0] <= ttft_slo and m[1] <= tpot_slo for m in map(metrics, traces)]
    return sum(ok) / len(ok)  # a request counts only if it met EVERY SLO

Predict: some benchmark tools compute TPOT as E2E divided by output tokens, which folds TTFT in. Is that number higher or lower than this definition for a request with a long queue wait?

Watch out

Two load-testing tools can report different TPOT on the same run because they disagree on whether the first token counts. Check the definition before comparing numbers across tools or vendors.

Use the library

In vLLM, all of this is built in: continuous batching and paged KV memory are always on, and prefix caching is on by default in recent versions (enable_prefix_caching). The knobs you actually tune are gpu_memory_utilization (how much memory goes to weights plus the block pool), max_num_seqs (batch slots), max_num_batched_tokens (the per-step token budget that chunked prefill splits long prompts into), and block_size. SGLang offers the same pieces with a radix-tree prefix cache and cache-aware scheduling.

Measuring a real server: most open-source serving engines expose an OpenAI-compatible HTTP API, so you can measure TTFT and TPOT with any streaming client.

import time, httpx, json

def measure(url, model, prompt, max_tokens=256):
    t0 = time.perf_counter(); times = []
    body = {"model": model, "prompt": prompt, "max_tokens": max_tokens, "stream": True}
    with httpx.stream("POST", f"{url}/v1/completions", json=body, timeout=120) as r:
        for line in r.iter_lines():
            if line.startswith("data: ") and line != "data: [DONE]":
                times.append(time.perf_counter())            # one chunk is usually one token
    ttft = times[0] - t0
    tpot = (times[-1] - times[0]) / max(1, len(times) - 1)
    return ttft, tpot

Run it from many concurrent clients at increasing request rates, and plot p50 and p95 TTFT and TPOT against throughput. That curve, not a single benchmark number, tells you how much traffic a replica can take within your SLO.

Run it

From guide/:

uv run code/18-llm-inference/serving_scratch.py

[paged] 4000 blocks: admitted 116/200 requests, 3996 blocks used, waste 781 tokens
[paged] contiguous 4096-token slabs fit 15 requests
[prefix] 50 requests: 98000/110000 prompt tokens from cache (89.1%)
[metrics] TTFT p50 102 ms, p99 729 ms
[metrics] TPOT p50 7.6 ms, p99 8.9 ms
[metrics] E2E p50 1.48 s
[goodput] TTFT<=500 ms and TPOT<=15 ms: 96.2%
[goodput] TTFT<=300 ms and TPOT<=9 ms: 85.3%
[goodput] TTFT<=300 ms and TPOT<=8 ms: 71.1%

Paged blocks admit 116 of 200 synthetic requests in a pool that would hold only 15 max-length slabs, wasting 781 token slots in total. All 49 requests after the first reuse the 2,000-token system prompt: $49 \cdot 2000 / 110{,}000 = 89.1\%$ of prompt tokens come from cache. Tightening TPOT from 9 ms to 8 ms barely moves the median, yet it drops goodput by 14 points, because SLOs bite in the tail.

The batching simulator (worked example, paged memory, prefix caching, cost, load sweep, and the figure) is code/18-llm-inference/batching_sim.py: uv run code/18-llm-inference/batching_sim.py.

5. Engineering

SLOs. Write them per use case, as percentiles: for a chat product, for example, p95 TTFT under 1 s and p95 TPOT under 50 ms (20 tokens/s, faster than people read). Batch jobs (offline summarization, evals, synthetic data) have no TTFT SLO at all, so run them at the largest batch that fits and on spare capacity.

The throughput-latency knob. Bigger batches raise tokens/s per GPU and cost less per token, but every step reads more KV and does more work, so TPOT rises. Most engines expose it as a maximum number of concurrent sequences and a maximum number of tokens per scheduler step. Tune both against the SLO curve from your load test.

Admission and fairness. Reject or queue early with a clear error rather than accepting work you cannot finish within SLO. Cap per-tenant concurrency so one customer's batch job can't starve interactive traffic. Route very long prompts to a separate pool.

Autoscaling signals. CPU and even GPU utilization are poor signals: a decode-bound GPU looks "busy" at any batch size. Scale on queue depth, KV-cache utilization, and measured TTFT and TPOT. Model weights take minutes to load, so keep warm capacity for peaks.

Prefix caching in practice. Put stable content (system prompt, tool definitions, few-shot examples, long documents) at the start of the prompt and the variable part at the end. A timestamp in the first line kills every cache hit. Route requests that share a prefix to the same replica.

Self-host or API. Self-hosting wins on steady high volume, data-residency or privacy requirements, custom fine-tuned models, and control over latency. An API wins for spiky or low volume, the newest frontier models, and teams without inference engineers. Many teams use both, with a router (production architecture).

Watch out

Failure modes. Preemption storms when long requests exhaust KV blocks (throughput collapses while GPUs look busy); a long-prompt request stalling every user's decode (fix with chunked prefill); prefix caches that never hit because of a per-request token at the start of the prompt; cross-tenant leakage if cached prefixes aren't isolated where required; tensor parallelism across slow links where communication eats the gain; load tests with fixed prompt and output lengths that hide the real length distribution.

Common mistakes

• Reporting average latency. Users feel p95 and p99, and queueing makes the tail grow much faster than the mean.
• Benchmarking with one prompt length and one output length. Batching gains depend on the spread.
• Sizing GPUs from request counts instead of prompt and output tokens.
• Pricing self-hosting at 100% utilization.
• Treating TTFT and TPOT as one latency. Queueing and prefill set TTFT; batch size and KV reads set TPOT.

6. Knowledge check

Practice exercise. Two slots, four requests all arriving at step 0 with output lengths $[4, 1, 1, 4]$. Compute the finish steps, throughput, idle %, and mean latency under static and continuous batching.

Implementation challenge. Extend the simulator with prefill. Give each request a prompt length $p_i$; prefill takes $\lceil p_i / C \rceil$ steps in its slot (chunk size $C$ tokens per step) and emits the first token at the end. Report mean and p95 TTFT as well as end-to-end latency. Then compare $C = 256$ with unchunked prefill, modelled as every other slot's decode pausing for $\lceil p_i / 256 \rceil$ steps while the whole prompt is processed.

Summary

• Continuous batching refills slots every step, so idle capacity falls and throughput and latency both improve; the gain grows with output-length spread.
• Paged KV memory allocates small blocks on demand, packing several times more sequences into the same memory, and makes prefix sharing possible.
• Prefix caching skips prefill for shared prompt prefixes; chunked prefill and disaggregation stop long prompts from stalling decode.
• Use data parallel replicas for throughput, tensor parallel when the model doesn't fit or latency is too high, pipeline and expert parallel for the largest models.
• Set SLOs on p95 TTFT and TPOT, load-test with realistic length distributions, and price capacity at real utilization.

Next: Tool use and agents

Related: LLM inference · Reliability, cost, and observability · The RAG pipeline · Cheat sheet: LLM inference

Go deeper, code-first (AI Engineering from Scratch): Real-Time Vision — Edge Deployment · GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling · Cold Start Mitigation for Serverless LLMs · Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d · Load Testing LLM APIs — Why k6 and Locust Lie · Self-Hosted Serving Selection — Matching Engine to Hardware and Scale

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill, Prefix-Cache Serving — RadixAttention and KV Reuse, Inference Metrics — TTFT, TPOT, ITL, Goodput, P99.

Interview angle