Adapting LLMs

1. Intuition

A pretrained base model continues text. To make it useful:

• Prompting: describe the task, give examples (few-shot), and specify the output format. Fastest; no training. Limited by context length and by what the model already knows.
• RAG: retrieve relevant documents at query time and put them in the prompt. Adds fresh or private knowledge and enables citations without retraining.
• Instruction tuning (supervised fine-tuning): train on (instruction, good response) pairs so the model follows instructions. This is how base models become assistants.
• Full fine-tuning: update all weights on your task data. Most powerful, most expensive, risks forgetting general skills.
• LoRA / PEFT (parameter-efficient fine-tuning): freeze the base weights and train small add-on matrices. Most of the benefit of fine-tuning at a fraction of the memory and storage.
• Preference optimization (RLHF, DPO): train on pairs of responses ranked by humans (or models) so the model prefers helpful, harmless, honest answers. Shapes behavior and style.

Hallucinations are fluent but unsupported claims. They follow from the objective: the model produces likely text, and a plausible-sounding answer is likely even when false.

2. Visualization

Illustrative sizes: LoRA with rank 8 on the four attention projections of 32 layers with width 4096 trains about 8.4 million parameters, about 0.1% of a 7B model.

3. The math

Symbols

LoRA

$B$ starts at zero, so training begins exactly at the pretrained model. After training, $BA$ can be merged into $W_0$, adding no inference latency, or kept separate so many small adapters can share one base model.

Supervised fine-tuning loss

The same next-token cross-entropy as pretraining, computed only on the response tokens of (instruction, response) pairs.

Direct preference optimization (DPO), intuition

Given a prompt with a preferred response $y_w$ and a rejected one $y_l$, increase the model's log-probability of $y_w$ relative to $y_l$, measured against a frozen reference model so the policy doesn't drift too far. It reaches a similar goal to RLHF without training a separate reward model or running reinforcement learning.

Worked example: LoRA savings

One attention projection is $4096 \times 4096$: $16{,}777{,}216$ parameters. LoRA with $r = 8$ trains $8 \times (4096 + 4096) = 65{,}536$, which is $0.39\%$ of that matrix. Applied to Q, K, V, and O in 32 layers: $65{,}536 \times 4 \times 32 = 8{,}388{,}608$ trainable parameters, about 0.12% of a 7-billion-parameter model. Storing that adapter in 16-bit precision takes about 17 MB instead of about 14 GB for a full copy.

Worked example: choosing an approach

4. Implementation

Prompting and RAG need no training code; RAG has its own lesson. What hides the most mechanism is preference tuning, so the build below covers it: code/15-llms/preference_scratch.py (NumPy only) checks the LoRA arithmetic, trains a reward model on preference pairs, and trains a toy policy with DPO. The policy is a softmax over four whole responses instead of a token-by-token LLM, which keeps every gradient to one line while the loss stays exactly DPO's.

Build it from scratch

Step 1: LoRA's parameter count. $A$ is $[r, d_{\text{in}}]$ and $B$ is $[d_{\text{out}}, r]$, so the adapter trains $r(d_{\text{in}} + d_{\text{out}})$ numbers instead of $d_{\text{in}} d_{\text{out}}$.

def lora_params(d_in, d_out, r):
    return r * (d_in + d_out)                     # A: [r, d_in] plus B: [d_out, r]

Predict: doubling $r$ from 8 to 16 doubles the adapter. What does doubling $d$ from 4096 to 8192 do to the adapter, and to the full matrix?

Step 2: preference data. Raters see two responses and pick one. They are noisy, so this script simulates them with the Bradley-Terry model: the better response wins with probability $\sigma(s_i - s_j)$, where $s$ is a hidden quality score (synthetic: 2, 1, 0, −1).

RESPONSES = ["cites the policy and answers", "answers, no citation", "vague hedge", "confident but wrong"]
TRUE_SCORE = np.array([2.0, 1.0, 0.0, -1.0])     # hidden quality, synthetic

def sample_pairs(n, rng):
    pairs = []
    for _ in range(n):
        i, j = rng.choice(4, size=2, replace=False)
        p_i = 1 / (1 + np.exp(-(TRUE_SCORE[i] - TRUE_SCORE[j])))   # Bradley-Terry: noisy raters
        pairs.append((i, j) if rng.random() < p_i else (j, i))     # (chosen, rejected)
    return np.array(pairs)

Predict: when "answers, no citation" (score 1) meets "vague hedge" (score 0), how often does the rater pick the worse one?

Step 3: the reward model, the first stage of RLHF. Learn a reward $r(y)$ so that $\sigma(r_w - r_l)$ matches the labels: logistic regression on the reward margin. Here each response gets one free scalar; a real reward model is an LLM with a scalar head, trained with this same loss.

def train_reward_model(pairs, steps=2000, lr=0.5):
    r = np.zeros(4)                               # one scalar reward per response (a real RM is a network)
    w, l = pairs[:, 0], pairs[:, 1]
    for _ in range(steps):
        g = sigmoid(r[w] - r[l]) - 1              # d/dmargin of -log sigmoid(r_w - r_l)
        grad = np.zeros(4)
        np.add.at(grad, w, g); np.add.at(grad, l, -g)
        r -= lr * grad / len(pairs)
        r -= r.mean()                             # rewards are only defined up to a constant
    return r

Predict: add 10 to every reward. Does the loss change?

Step 4: the DPO loss. DPO skips the reward model. It scores each response by how much more likely the policy makes it than the frozen reference does (the log-ratio), and applies the same logistic loss to the gap, scaled by $\beta$.

def dpo_loss(chosen_logratio, rejected_logratio, beta):
    """Each log-ratio is log pi_theta(y|x) - log pi_ref(y|x), summed over the response's tokens."""
    return -np.log(sigmoid(beta * (chosen_logratio - rejected_logratio)))

At step 0 the policy equals the reference, both log-ratios are 0, and the loss is $-\ln 0.5 = 0.693$ for every pair. With log-ratios $+2.0$ and $-1.0$ at $\beta = 0.1$, it is $0.554$, the numbers worked in fine-tuning in practice.

Predict: if the chosen and rejected log-ratios both fall by 5, what happens to the loss?

Step 5: DPO training. The policy is logits $\theta$ over the four responses. The gradient of $\log \pi(y)$ with respect to $\theta$ is $\text{onehot}(y) - \pi$; in a chosen-minus-rejected difference the $\pi$ terms cancel, which is why the update is so short.

def train_dpo(pairs, beta, steps=3000, lr=1.0):
    theta = np.zeros(4)                           # policy logits; start equal to the reference
    ref = log_softmax(theta)                      # frozen reference log-probs
    w, l = pairs[:, 0], pairs[:, 1]
    for _ in range(steps):
        lp = log_softmax(theta)
        h = beta * ((lp[w] - ref[w]) - (lp[l] - ref[l]))
        g = beta * (sigmoid(h) - 1)               # d loss / d (log pi_w - log pi_l)
        grad = np.zeros(4)                        # d log pi_w / d theta = onehot_w - pi; the pi terms cancel
        np.add.at(grad, w, g); np.add.at(grad, l, -g)
        theta -= lr * grad / len(pairs)
    return np.exp(log_softmax(theta))

The script then checks DPO against the answer RLHF is aiming for. RLHF maximizes reward minus $\beta$ times the KL divergence from the reference, and that problem has a closed-form best policy:

DPO is derived from this formula, so training on the pairs should land on it without ever building $r$.

Predict: which $\beta$, 0.5 or 2.0, moves the policy further from the uniform reference?

Use the library

Few-shot prompting needs only messages; LoRA and DPO come from Hugging Face PEFT and TRL:

messages = [
    {"role": "system", "content": "Classify the ticket. Reply with JSON: {\"category\": ..., \"urgency\": 1-5}."},
    {"role": "user", "content": "Ticket: 'Cannot log in since update.'"},
    {"role": "assistant", "content": "{\"category\": \"auth\", \"urgency\": 4}"},
    {"role": "user", "content": f"Ticket: '{ticket}'"},
]

from peft import LoraConfig, get_peft_model        # pip install peft transformers trl
config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], lora_dropout=0.05)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()                  # e.g. "trainable params: 8,388,608 || all params: ~6.7B || trainable%: ~0.12"

TRL's DPOTrainer takes a dataset of {"prompt", "chosen", "rejected"} rows and computes step 4 with real token log-probabilities: it sums $\log \pi$ over each response's tokens for the policy and the frozen reference, usually precomputing the reference once. RewardTrainer implements step 3 on an LLM with a scalar head. The LoRA wrapper and the full SFT and memory story are in fine-tuning in practice.

Run it

uv run code/15-llms/preference_scratch.py

Key lines of output (synthetic preference data):

Q,K,V,O x 32 layers: 8,388,608 trainable; bf16 adapter 16.8 MB
step 0 (policy = reference): 0.693
chosen +2.0, rejected -1.0 : 0.554
learned rewards: [ 1.56  0.54 -0.59 -1.51]  (true scores minus their mean: [ 1.5  0.5 -0.5 -1.5] )
reward model agrees with 82.3% of rater labels (raters are noisy by design)
beta=0.5: DPO [0.874 0.112 0.012 0.002]  closed form [0.874 0.112 0.012 0.002]
beta=2.0: DPO [0.464 0.278 0.158 0.1  ]  closed form [0.464 0.278 0.158 0.1  ]

The reward model recovers the hidden scores from noisy labels; 82% agreement is the ceiling the raters' own noise allows, not a training failure. DPO lands on the closed-form RLHF policy to three decimals without a reward model, and smaller $\beta$ concentrates mass on the best response more aggressively.

5. Engineering

Decision order. Try prompting first (cheapest, fastest to iterate). Add RAG when the model lacks knowledge or you need citations. Fine-tune (usually LoRA) when you need consistent behavior, format, or domain style that prompting can't reliably produce, or to make a smaller, cheaper model match a larger one on a narrow task. Every step needs an evaluation set to prove it helped.

Fine-tuning data. Quality beats quantity: a few thousand clean, representative examples often beat a large noisy set. Hold out evaluation data. Watch for regressions on general capabilities (catastrophic forgetting) and safety behavior.

Preference tuning in practice. RLHF (reward model, then PPO against it with a KL penalty) is powerful but runs several models at once and is touchy to tune. DPO optimizes the same objective directly from pairs and is the usual starting point; variants change the loss (IPO adds a margin target against overfitting, KTO learns from single thumbs-up or thumbs-down labels instead of pairs, ORPO and SimPO drop the reference model). Two failure modes to watch: reward hacking, where the policy finds what the reward model over-scores (length, confident tone) rather than what raters wanted, and DPO's tendency to push the chosen response's probability down too, as long as the rejected one falls faster.

Hallucination mitigations that work: grounding answers in retrieved sources with citations; tool calls for facts that live in systems (prices, inventory, calculations); constrained output formats; verification steps (check claims against sources); allowing and rewarding "I don't know"; human review for consequential decisions; evaluation on realistic, adversarial cases.

Uncertainty. Token probabilities reflect how likely the text is, not whether a claim is true. Verbal confidence ("I'm certain") is not calibrated. Measure calibration empirically if you plan to use confidence for routing.

Key distinction

RAG changes the model's context; fine-tuning changes its weights. RAG is usually preferable for private or frequently changing facts (update the index, cite sources). Fine-tuning is useful for changing behavior, format, or specialization, and is not a reliable way to inject facts that change.

Watch out

Failure modes. Fine-tuning to teach facts and getting confident hallucinations of those facts; overfitting a small fine-tuning set; losing safety behavior after fine-tuning; prompt changes shipped without evaluation; mistaking fluency for correctness.

Common mistakes

• Reaching for fine-tuning before trying a well-engineered prompt and RAG.
• Evaluating adaptations by reading a handful of outputs.
• Assuming lower temperature reduces hallucinations meaningfully.

6. Knowledge check

Practice exercise. A layer has $d_{\text{in}} = 2048$ and $d_{\text{out}} = 8192$. How many parameters does LoRA with $r = 16$ train, and what fraction of the full matrix is that?

Implementation challenge. Pick a small open model and a narrow task (for example, rewriting support replies in a fixed style). Build a 50-example evaluation set, measure a zero-shot prompt, a few-shot prompt, and a LoRA fine-tune, and report quality and cost per request for each.

Summary

• Prompting and RAG change context; fine-tuning, LoRA, and preference optimization change weights.
• LoRA trains low-rank updates $BA$ with $r(d_{\text{in}} + d_{\text{out}})$ parameters, often well under 1% of the model.
• Instruction tuning creates assistants; preference optimization (RLHF, DPO) shapes behavior.
• Use RAG or tools for changing facts, fine-tuning for behavior and format, and always evaluate.
• Hallucinations come from the objective; grounding, tools, verification, and abstention reduce them.

Next: Fine-tuning in practice

Related: LLM evaluation · Transfer learning · Model card: large language model

Go deeper, code-first (AI Engineering from Scratch): Constitutional AI and Self-Improvement · STaR, V-STaR, Quiet-STaR — Self-Taught Reasoning · Reward Hacking and Goodhart's Law · Sycophancy as RLHF Amplification

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Reward Modeling & RLHF, RLHF: Reward Model + PPO, DPO: Direct Preference Optimization, The Direct Preference Optimization Family, Capstone Lesson 40: Direct Preference Optimization from Scratch.

Interview angle