Skip to content
ML to LLMs
Labs / LLM and RAG Lab

Temperature, top-k, top-p

Turn logits into a next-token distribution and watch each sampling knob reshape it.

Theory and worked examples: Decoding

Decoding lab

Prompt: “The capital of France is …”. Synthetic logits for 10 candidate tokens (illustrative, not from a real model).

“Paris”0.854z=6.1
“a”0.047z=3.2
“the”0.035z=2.9
“located”0.021z=2.4
“known”0.016z=2.1
“Lyon”0.009z=1.6
“not”0.006z=1.2
“beautiful”0.005z=1
“France”0.003z=0.6
“Marseille”0.003z=0.4

Gray: softmax with temperature. Blue: after top-k and top-p filtering and renormalization.

Pipeline: logits→  /T  softmax→top-k→top-prenormalize→sample\text{logits} \xrightarrow{\;/T\;} \text{softmax} \xrightarrow{\text{top-}k} \xrightarrow{\text{top-}p} \text{renormalize} \to \text{sample}

At T = 1 the model's own distribution is used unchanged. These probabilities describe which token tends to come next in text, not whether a statement is true. A fluent wrong answer can have high probability.