Fine-tuning in practice

1. Intuition

The post-training pipeline. Modern assistants are built in stages. Each stage uses a different kind of data and a different loss.

1. Pretraining. Next-token prediction on trillions of tokens of web, code, and books. The model learns language, facts, and reasoning patterns. It produces continuations, not answers.
2. Supervised fine-tuning (SFT). Train on (prompt, ideal response) pairs, often tens of thousands to a few million. The model learns the assistant format: follow instructions, use the chat template, stop at the right place.
3. Preference tuning. Train on (prompt, better response, worse response) triples. The model learns what humans (or a judge model) prefer: helpfulness, tone, refusals. RLHF with a reward model and PPO was the original recipe. DPO and its variants do it with a simple classification-style loss.
4. Reinforcement learning with verifiable rewards (RLVR). For tasks with a checkable answer (math with a known result, code with unit tests, a JSON output that must parse), sample several answers, score them with a program, and push up the probability of the ones that pass. GRPO is a popular algorithm here. This stage is where much of the recent progress on step-by-step reasoning has come from.

As an application engineer you usually start from a model that has already been through all four stages. Your own fine-tuning is typically a small SFT run, sometimes followed by DPO on your preference data, sometimes RL against your own verifier.

SFT data is a product. The model learns exactly what the examples show, including their mistakes. A few thousand clean, diverse, correctly formatted examples usually beat a much larger set of noisy ones. Duplicates teach the model to overweight those cases. Examples that overlap your evaluation set make your scores meaningless (contamination).

Why memory is the first wall. Training needs far more memory than inference. For every weight you train, the optimizer keeps extra state. Full fine-tuning of a 7B model needs roughly 16 bytes per parameter before activations: over 100 GB. LoRA freezes the base model and trains small add-on matrices, so the extra state shrinks to well under 1 GB. QLoRA also stores the frozen base in 4 bits, so a 7B model fits on a 24 GB consumer GPU.

Forgetting. Push a model hard on a narrow task and it can lose skills it had before: general reasoning, other languages, safety behavior. This is catastrophic forgetting. You only notice it if you test for it.

2. Visualization

Try it (predict first, then check):

1. On the 7B preset, compare the full fine-tune bar with the LoRA bar below it. Predict which segment shrinks most. Then check whether it was weights, gradients, or optimizer states.
2. Double the LoRA rank from 16 to 32. Predict the change in trainable parameters and in total memory. Why does total memory barely move?
3. Pick the 70B preset with the 48 GB GPU line. Predict which of the three methods fits. Then turn off gradient checkpointing and raise the tokens per micro-batch.
4. Change the targets from "all linear" to "q, v only". Predict the drop in trainable parameters for the 7B preset before you look.

3. The math

Symbols

SFT loss with masking

An SFT example is one sequence: the prompt rendered in the model's chat template (special tokens that mark system, user, and assistant turns) followed by the response. The loss is next-token cross-entropy, but only on response tokens:

With a 300-token prompt and a 100-token response, the mask has 300 zeros then 100 ones, so the loss averages over 100 tokens. Without the mask, three quarters of the gradient would teach the model to write user prompts. In PyTorch libraries the mask is implemented by setting the label to -100 for prompt positions.

Full fine-tuning memory

With mixed-precision AdamW, each trained parameter costs:

Plus activations, which scale with tokens per micro-batch rather than with $N$. Optimizer states are 12 of the 16 bytes. That is why sharding them across GPUs (ZeRO, FSDP) or storing them in 8 bits are the main levers for full fine-tuning.

LoRA

• Initialization. $A$ is random and $B$ is zero, so $BA = 0$ and step 0 is exactly the pretrained model. Gradients still flow into $B$ because $A$ is non-zero.
• Scaling. $\alpha/r$ keeps the update's size roughly stable when you change $r$, so you can change rank without fully retuning the learning rate. Common settings are $\alpha = r$ or $\alpha = 2r$. (rsLoRA uses $\alpha/\sqrt{r}$ for high ranks.)
• Which matrices. Each transformer layer has seven linear maps: attention $q, k, v, o$ and MLP gate, up, down. Targeting all of them usually adapts better than attention-only, at a few times the trainable parameters.
• Merging. At deployment, compute $W_0 + \frac{\alpha}{r}BA$ once. The merged model has the same shape and speed as the base. Or keep adapters separate and hot-swap many small adapters on one shared base.

QLoRA

QLoRA stores the frozen base weights in NF4, a 4-bit format whose 16 levels are placed at quantiles of a normal distribution (pretrained weights are roughly normal). Each block of 64 weights shares one scale; double quantization compresses those scales too, for about 4.127 bits per weight in total. In the forward pass, each block is dequantized to bf16 on the fly and multiplied. The LoRA matrices $A, B$ stay in 16 bits and are the only things trained. Gradients flow through the quantized base to reach them.

Worked example: a synthetic 7B-style model

Configuration (synthetic, Llama-like shape): $d = 4096$, $L = 32$, $d_{ff} = 11008$, vocabulary 32000, full multi-head attention so $k$ and $v$ are $4096 \times 4096$.

Step 1: parameters per layer.

Per layer: $4 \times 16{,}777{,}216 + 3 \times 45{,}088{,}768 = 67{,}108{,}864 + 135{,}266{,}304 = 202{,}375{,}168$. All layers: $202{,}375{,}168 \times 32 = 6{,}476{,}005{,}376$. Embedding plus untied output head: $2 \times 32000 \times 4096 = 262{,}144{,}000$. Total: $N = 6{,}476{,}005{,}376 + 262{,}144{,}000 = 6{,}738{,}149{,}376 \approx 6.74$ billion. (Norm weights add about 0.3 million; ignored.)

Step 2: LoRA with $r = 16$ on all seven linear layers.

• $q, k, v, o$: $16 \times (4096 + 4096) = 131{,}072$ each, so $4 \times 131{,}072 = 524{,}288$.
• gate, up, down: $16 \times (4096 + 11008) = 16 \times 15{,}104 = 241{,}664$ each, so $3 \times 241{,}664 = 724{,}992$.
• Per layer: $524{,}288 + 724{,}992 = 1{,}249{,}280$. All layers: $1{,}249{,}280 \times 32 = 39{,}976{,}960$.
• Share of the model: $39{,}976{,}960 / 6{,}738{,}149{,}376 = 0.593\%$.
• Adapter file in bf16: $39{,}976{,}960 \times 2 = 79{,}953{,}920$ bytes $\approx 80$ MB.

(Attention-only with $r = 8$, the earlier lesson's example, gives $8 \times 8192 \times 4 \times 32 = 8{,}388{,}608$, or 0.124%.)

Step 3: activations (rough). A common estimate for one layer's saved activations in 16-bit, with a fused attention kernel, is about $34 \cdot s \cdot d$ bytes for $s$ tokens. For $s = 2048$: $34 \times 2048 \times 4096 = 285{,}212{,}672$ bytes per layer. Across 32 layers without checkpointing: $9.13$ GB. With gradient checkpointing we keep only each layer's input ($2 \cdot s \cdot d = 16{,}777{,}216$ bytes, times 32 $= 536{,}870{,}912$) plus one layer's full activations during recompute ($285{,}212{,}672$): $822{,}083{,}584$ bytes $\approx 0.82$ GB.

Step 4: memory per method (decimal GB, checkpointing on).

where $T = 39{,}976{,}960$ trainable LoRA parameters. Check the full fine-tuning total: $16 \times 6{,}738{,}149{,}376 = 107{,}810{,}390{,}016$ bytes, plus $822{,}083{,}584$, gives $108{,}632{,}473{,}600$ bytes $= 108.63$ GB. Full fine-tuning does not fit on one 80 GB GPU. LoRA fits on a 24 GB GPU with room for larger batches. QLoRA fits on a 24 GB GPU with a lot of headroom, or lets a much larger model fit on a 48 GB or 80 GB GPU.

Note

Rank barely matters for memory. Doubling $r$ to 32 doubles $T$ to 79.95 million, which adds $16 \times 39{,}976{,}960 \approx 0.64$ GB. The frozen base dominates. That is why QLoRA's 4-bit base, not a smaller rank, is what moves the bar.

DPO: preference tuning without a reward model

The RLHF recipe trains a reward model on preference pairs, then uses PPO to maximize reward with a KL penalty that keeps the policy near a reference model. PPO needs four models in memory (policy, reference, reward, value) and is fiddly to tune. DPO shows that the same objective can be optimized directly from the pairs:

Each log-probability is summed over the response's tokens. Intuition: raise the chosen response's probability relative to where the reference model had it, and lower the rejected one's. $\beta$ controls how far the policy may move: small $\beta$ allows bigger moves.

Worked numbers with $\beta = 0.1$:

• At step 0 the policy equals the reference, so both log-ratios are 0. The argument is $0$, $\sigma(0) = 0.5$, loss $= -\ln 0.5 = 0.693$.
• Later, chosen log-ratio $= +2.0$ and rejected $= -1.0$. Argument $= 0.1 \times (2.0 - (-1.0)) = 0.3$. $\sigma(0.3) = 1/(1 + e^{-0.3}) = 1/(1 + 0.7408) = 0.5744$. Loss $= -\ln 0.5744 = 0.554$.

DPO only needs the policy and a frozen reference (whose log-probs can be precomputed). A known failure: both log-ratios can fall together while their gap grows, so the chosen response's absolute probability drops. Monitor both, not just the loss.

GRPO and RL with verifiable rewards

GRPO (group relative policy optimization) drops PPO's value network. For each prompt it samples a group of $G$ responses, scores each with a reward $r_i$ (often a program: did the tests pass?), and uses the group-normalized reward as the advantage:

Example: $G = 4$, rewards $[1, 0, 0, 1]$. Mean $= 0.5$. Standard deviation $= \sqrt{\frac{1}{4}(0.25 + 0.25 + 0.25 + 0.25)} = 0.5$. Advantages $= [1, -1, -1, 1]$. Tokens of passing answers are pushed up, failing ones pushed down, with a PPO-style clipped ratio and a KL term to the reference. If all four pass or all four fail, the standard deviation is 0 and the prompt gives no learning signal, so prompt difficulty matters.

Distillation

Distillation trains a small student to imitate a large teacher. Sequence-level distillation is just SFT on teacher-generated responses. Logit-level distillation minimizes the KL divergence between teacher and student token distributions, which carries more information per token but requires a shared tokenizer and access to the teacher's logits. Distillation is the usual way to make a small, cheap model match a big one on a narrow task.

4. Implementation

The full script is code/15-llms/finetune_scratch.py (NumPy only, no GPU). It builds the three mechanisms every fine-tuning library hides: the SFT loss mask, a LoRA layer with its own backward pass and merge, and gradient accumulation done right.

Build it from scratch

Step 1: the loss mask. The model still predicts every next token, but only response positions count. Dividing by mask.sum() instead of the sequence length makes the loss an average over response tokens, matching the formula in section 3.

def masked_nll(logp_true, mask):
    """logp_true: log-prob of each actual next token [T]; mask: 1 on response tokens, 0 on prompt."""
    return -(logp_true * mask).sum() / mask.sum()   # average over response tokens only

The script uses section 3's shape: 300 prompt tokens the model finds hard (probability 0.05 each) and 100 response tokens at 0.5. Masked, the loss is $-\ln 0.5 = 0.693$. Unmasked it is $(300 \times 2.996 + 100 \times 0.693)/400 = 2.420$, and three quarters of the gradient would go into predicting the user's prompt.

Predict: if you divided by the sequence length (400) but kept the mask, what loss would you report, and why is that misleading when responses vary in length?

Step 2: a LoRA layer. The frozen $W_0$ plus a low-rank path $\frac{\alpha}{r} B A$. $B$ starts at zero so step 0 reproduces the base model exactly; $A$ is random so gradients can reach $B$. merged() folds the adapter back into one matrix.

class LoRALinear:
    def __init__(self, W0, r, alpha, rng):
        d_out, d_in = W0.shape
        self.W0 = W0                                    # frozen
        self.A = rng.normal(0, 1 / np.sqrt(d_in), (r, d_in))   # random
        self.B = np.zeros((d_out, r))                   # zero, so B @ A = 0 at step 0
        self.s = alpha / r

    def forward(self, X):                               # X: [n, d_in]
        self.X, self.XA = X, X @ self.A.T               # [n, r]: the low-rank bottleneck
        return X @ self.W0.T + self.s * self.XA @ self.B.T

    def merged(self):
        return self.W0 + self.s * self.B @ self.A       # same shape as W0: no extra latency

Predict: what goes wrong if both $A$ and $B$ start at zero?

Step 3: LoRA's backward pass. Only $A$ and $B$ get gradients; $W_0$ is never touched, which is where LoRA's memory savings come from (no gradient or optimizer state for $W_0$). The chain rule runs through the bottleneck XA.

def lora_step(layer, X, Y, lr):
    pred = layer.forward(X)
    dP = 2 * (pred - Y) / len(X)                        # mean squared error gradient, [n, d_out]
    dB = layer.s * dP.T @ layer.XA                      # [d_out, r]
    dA = layer.s * (dP @ layer.B).T @ layer.X           # [r, d_in]  (zero while B is zero)
    layer.B -= lr * dB
    layer.A -= lr * dA
    return ((pred - Y) ** 2).sum(1).mean()

The demo gives a 64×64 layer a synthetic task that needs a rank-4 change to its weights. LoRA with $r = 4$ trains 512 numbers (12.5% of 4,096) and fits it.

Predict: if the task needed a rank-16 change and you kept $r = 4$, would the loss reach zero?

Step 4: gradient accumulation. When a batch does not fit in memory, run several micro-batches, add their gradients, and step once. The trap in SFT: with loss masking, micro-batches hold different numbers of response tokens. Averaging each micro-batch's mean weights a 1-token micro-batch as much as an 8-token one. Sum the gradients, then divide by the total token count.

def grad_sum(w, X, y, mask):
    """Gradient of the SUM of masked squared errors for a linear model, and the token count."""
    err = (X @ w - y) * mask
    return 2 * X.T @ err, mask.sum()

parts = [grad_sum(w, Xb[i:i + 8], yb[i:i + 8], mb[i:i + 8]) for i in range(0, 32, 8)]
naive = sum(g / n for g, n in parts) / 4                        # mean of per-micro-batch means
right = sum(g for g, _ in parts) / sum(n for _, n in parts)     # sum, then divide by total tokens

A linear model stands in for the LLM so the gradient is one line; the bookkeeping is identical.

Predict: if every micro-batch had exactly the same number of response tokens, would naive and right agree?

Use the library

The production version of all four steps, with QLoRA:

# SFT with loss masking and QLoRA (pip install transformers peft bitsandbytes trl)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

tok = AutoTokenizer.from_pretrained(BASE)
def encode(ex):
    prompt = tok.apply_chat_template(ex["messages"][:-1], tokenize=False, add_generation_prompt=True)
    full = prompt + ex["messages"][-1]["content"] + tok.eos_token
    ids = tok(full, add_special_tokens=False)["input_ids"]
    n_prompt = len(tok(prompt, add_special_tokens=False)["input_ids"])
    return {"input_ids": ids, "labels": [-100] * n_prompt + ids[n_prompt:]}  # mask prompt tokens

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb)
model = prepare_model_for_kbit_training(model)  # enables gradient checkpointing
model = get_peft_model(model, LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
model.print_trainable_parameters()

The -100 labels are step 1's mask: PyTorch's cross-entropy ignores that label. PEFT wraps each target nn.Linear in step 2's layer, and autograd replaces step 3. gradient_accumulation_steps in the Trainer is step 4; recent versions of Hugging Face Transformers normalize by the total token count across micro-batches, after older versions had exactly the averaging bug shown above. Libraries such as TRL wrap the whole thing (SFTTrainer with assistant-only loss, DPOTrainer, GRPOTrainer). After training, model.merge_and_unload() is step 2's merged(): it folds the adapter into a 16-bit copy of the base for deployment.

Run it

uv run code/15-llms/finetune_scratch.py

Key lines of output (synthetic weights and data):

masked loss (response only): 0.693
unmasked loss (all tokens) : 2.420  <- dominated by the prompt
step 0 output equals base model: True
trainable 512 vs full 4,096 (12.5%)
step   0  loss 4.2323
step 400  loss 0.0019
merged weights give the same outputs: True
tokens per micro-batch: [8, 2, 8, 1]
naive  (average of means) matches full batch: False  max error 1.138
correct (divide by total) matches full batch: True

The memory arithmetic of section 3 (parameter counts, the per-method memory table, DPO and GRPO numbers, and the figure) is in code/15-llms/fine_tuning_memory.py.

5. Engineering

Data pipeline. Render every example with the exact chat template the model will see at inference (a mismatch between training and serving templates is a classic silent bug). Deduplicate exact and near-duplicate examples (for example MinHash over n-grams). Check contamination: remove training examples that overlap your eval set by long n-gram matches. Strip secrets and personal data. Read a random sample of 50 examples yourself before every run.

Hyperparameters that matter. For LoRA, learning rates are typically around $10^{-4}$ (higher than full fine-tuning, which is typically around $10^{-5}$), 1 to 3 epochs, warmup plus cosine or linear decay. More epochs on a small set overfits fast: the training loss keeps dropping while held-out quality and general skills fall. Pack multiple short examples into one sequence (with attention masks that keep them separate) to avoid wasting compute on padding.

Memory levers, in order. Gradient checkpointing (trades about one extra forward pass for most activation memory). LoRA instead of full fine-tuning. QLoRA's 4-bit base. Smaller micro-batch with gradient accumulation. For full fine-tuning: 8-bit optimizers, ZeRO/FSDP sharding across GPUs, CPU offload.

Regression evals. Before and after every run, score three suites: (1) the target task, (2) a general-capability suite (reasoning, instruction following, other languages you care about), (3) safety and refusal behavior. Ship only if (1) improves and (2) and (3) stay within a tolerance you set in advance. Mitigations for forgetting: mix a slice of general instruction data into training (replay), lower the learning rate, fewer epochs, LoRA with modest rank.

Serving. A merged model is a normal checkpoint. Unmerged adapters let one GPU serve many customers' fine-tunes (multi-LoRA serving batches requests for different adapters together). Adapters are tied to the exact base checkpoint: when the base model is upgraded, you must retrain them.

When fine-tuning is the wrong tool. Facts that change (use RAG or tools). Fewer than a few hundred good examples and no eval set. A behavior that a better prompt or structured outputs already gets right. Needing citations to sources. A base model your provider will deprecate in months.

Watch out

Failure modes. Loss computed on prompt tokens; train/serve chat-template mismatch; missing EOS token so the model never stops; eval examples leaked into training; overfitting from too many epochs; forgetting general skills or safety behavior; DPO lowering the chosen response's probability; QLoRA adapter merged into a 4-bit base and requantized, which loses quality (merge into a 16-bit copy instead).

Common mistakes

• Estimating training memory as "model size times 2" and forgetting gradients and optimizer states.
• Believing a lower LoRA rank is the main memory saver. The frozen base dominates.
• Comparing fine-tuned and base models on the training distribution only.
• Fine-tuning to add knowledge, then being surprised by confident wrong answers on facts just outside the training set.
• Mixing up DPO's $\beta$ with a learning rate. $\beta$ is the strength of the pull toward the reference.
• Running RL with a verifier that can be gamed (for example, tests that pass on empty output). The model will find the loophole.
• Averaging per-micro-batch losses under gradient accumulation when micro-batches hold different numbers of response tokens. Sum, then divide by the total token count (section 4, step 4).

6. Knowledge check

Practice exercise. A synthetic model has $d = 2048$, $L = 24$, $d_{ff} = 5504$, full multi-head attention. You apply LoRA with $r = 8$ to $q$ and $v$ only. How many trainable parameters? How much memory does full fine-tuning need for 2 billion total parameters (excluding activations)?

Implementation challenge. Write encode(example) for a two-turn conversation (user, assistant, user, assistant) that masks every token except the two assistant replies. Test it by decoding only the positions whose label is not -100 and checking you get exactly the two replies plus end-of-turn tokens.

Summary

• Post-training runs pretraining, then SFT, then preference tuning (RLHF or DPO), then RL with verifiable rewards (for example GRPO). Your fine-tune is usually a small SFT or DPO run on top.
• SFT trains next-token cross-entropy on response tokens only, using the serving chat template. Data quality, dedup, and contamination checks matter more than volume.
• Full fine-tuning with mixed-precision AdamW costs about 16 bytes per parameter plus activations: about 108.6 GB for the synthetic 7B-style model.
• LoRA trains $r(d_{\text{in}} + d_{\text{out}})$ parameters per matrix (0.59% at $r = 16$ on all linear layers), starts at $BA = 0$, and merges into $W_0$ for free inference. QLoRA stores the base in NF4 and brings the job to about 5.3 GB.
• DPO optimizes preferences directly against a frozen reference; GRPO normalizes verifier rewards within a group of samples. Always run regression evals for forgetting.

Next: Mixture of experts

Related: Adapting LLMs · Evaluating LLM systems · Training and regularization · Cheat sheet: fine-tuning

Go deeper, code-first (AI Engineering from Scratch): ControlNet, LoRA & Conditioning · Loading Pretrained Weights · Capstone Lesson 38: Classifier Fine-Tuning by Head Swap

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Instruction Tuning (SFT), Fine-Tuning with LoRA & QLoRA, Gradient Accumulation.

Interview angle