Context engineering

1. Intuition

Everything the model knows at answer time is in the window. A model call sees one sequence of tokens. It typically holds:

The context window limit covers input and output, so the output reserve comes out of the same budget.

Big windows are not free. Windows of hundreds of thousands of tokens, and some of a million or more, are common in 2026. But every input token costs money, adds prefill time before the first output token, and occupies KV cache memory on the server. And more text is not more understanding: measured accuracy on long-context tests generally drops as irrelevant text grows, and models often use information at the start and end of a long input better than information in the middle. That last effect is called lost in the middle (Liu et al., 2023, "Lost in the Middle: How Language Models Use Long Contexts"). Its size varies by model and task and has shrunk in newer models, but the safe assumption is still: less, better-placed context beats more context.

Four moves cover most of context engineering.

1. Select. Include only what this call needs: the top few relevant chunks, not the top twenty; the tools for this intent, not all of them.
2. Order. Stable material first (system prompt, tool schemas), so prefix caching can reuse it. The most relevant retrieved items at the edges of the retrieved block, the least relevant in the middle. The question last, close to where generation starts.
3. Compress. Replace old conversation turns with a summary; trim long tool outputs to the fields that matter; paginate large results.
4. Remember outside the window. Facts that must survive across sessions go into a store (files, a database, a vector index) and are retrieved when relevant. Long-term memory is just retrieval over your own notes.

Key distinction

Context engineering versus prompt engineering. Prompt engineering is how you phrase the instructions. Context engineering is everything that ends up in the window and where it sits: retrieval, tool selection, history management, memory, and ordering. In agents and RAG systems, the second matters more.

2. Visualization

Synthetic component sizes for a coding assistant (token counts estimated at 1.3 tokens per word). Left: one call uses under a tenth of the window. Right: what happens across a conversation.

The left panel is why "we have a big window" is the wrong question. The window is mostly empty; the problem is not space, it's what you pay for and what the model attends to. The right panel shows the naive approach (all 30 tools, 8 documents, full history) growing by one turn's worth of tokens every call, forever. The engineered approach (10 relevant tools, 3 documents, the last 4 turns plus a 150-token summary) stops growing once compaction kicks in.

Try it (predict first, then edit context_engineering.py):

1. Set KEEP = 10 in the engineered setup. Predict where the green line flattens and its height.
2. Double TURN to 500 tokens. Predict which line changes slope and by how much.
3. In the ordering demo, add a second document with score 1.0. Predict where edges-first ordering puts it.
4. Change the query in the tool-selection demo to mention "calendar" and "email". Predict the number of selected tools and tokens saved.

3. The math

Symbols

Budget

The input must fit in what's left after the reserve:

For the synthetic assistant: $W = 128{,}000$, $R = 4{,}000$, so the input budget is $124{,}000$. Components: $500 + 4{,}500 + 4{,}000 + 3{,}000 + 200 = 12{,}200$, which is $12{,}200 / 128{,}000 = 9.5\%$ of the window. Room left: $124{,}000 - 12{,}200 = 111{,}800$.

Per-call input, naive

Keeping full history, call $t$ re-reads $t - 1$ earlier turns:

This is the same arithmetic series as the agent loop: total billed input grows with $T^2$.

Per-call input, with compaction

Keeping the last $k$ turns plus an $m$-token summary:

Once $t > k + 1$, $c_t$ is constant, so the total grows linearly in $T$.

Worked example: 20 calls

Naive: 30 tools × 150 = 4,500; 8 documents × 500 = 4,000; system 500; query 200. $F = 9{,}200$. $h = 250$.

1. Call 1: $9{,}200$. Call 20: $9{,}200 + 250 \times 19 = 9{,}200 + 4{,}750 = 13{,}950$.
2. Total: $20 \times 9{,}200 + 250 \times \frac{20 \times 19}{2} = 184{,}000 + 250 \times 190 = 184{,}000 + 47{,}500 = 231{,}500$.

Engineered: 10 tools × 150 = 1,500; 3 documents × 500 = 1,500; system 500; query 200. $F = 3{,}700$. $k = 4$, $m = 150$.

3. Calls 1 to 5 (history 0, 250, 500, 750, 1,000): $5 \times 3{,}700 + 250 \times (0+1+2+3+4) = 18{,}500 + 2{,}500 = 21{,}000$.
4. Calls 6 to 20 (15 calls), each $3{,}700 + 250 \times 4 + 150 = 3{,}700 + 1{,}000 + 150 = 4{,}850$: $15 \times 4{,}850 = 72{,}750$.
5. Total: $21{,}000 + 72{,}750 = 93{,}750$, which is $93{,}750 / 231{,}500 = 0.40$ of the naive bill.

Selection (fewer tools and documents) cut the fixed part from 9,200 to 3,700. Compaction cut the growing part. Both changes also leave fewer distractors in the window.

Edges-first ordering

Sort items by relevance score, best first: $d_1, d_2, \dots, d_n$. Put the odd ranks at the front and the even ranks, reversed, at the back:

With six documents scored $[1.0, 0.4, 0.2, 0, 0, 0]$, the order is $1.0, 0.2, 0, 0, 0, 0.4$: the two best at the two edges, the weakest in the middle. The relevance score here is simple word overlap, the fraction of query words in the document: for "refund window for damaged items" and "damaged items ship back free", 2 of 5 query words match, so $0.4$.

4. Implementation

Build it from scratch

No model call is needed: the subject is what you would send. Start with a token counter. Real counts come from the model's tokenizer; 1.3 tokens per English word is a planning estimate.

def count_tokens(text):
    """Rough estimate. Use the provider's token-count endpoint or tokenizer in production."""
    return int(len(text.split()) * 1.3)

Predict: is this estimate likely to be high or low for source code full of symbols?

The budget object reserves output space up front and trims each component to the lesser of its cap and what is left. Components added first get space first, which is why you add the essentials first.

class ContextBudget:
    def __init__(self, window, reserve_output):
        self.available = window - reserve_output       # output tokens share the window
        self.used = {}

    def add(self, name, text, cap=None):
        limit = min(cap or 10**9, self.available - sum(self.used.values()))
        words = text.split()[: int(limit / 1.3)]         # keep the words that fit (1.3 tokens/word)
        self.used[name] = count_tokens(" ".join(words))
        return " ".join(words)

Predict: with an 8,000 window, 1,000 reserve, a 400-word system prompt, and 3,000 words of documents capped at 2,000 tokens, how many tokens remain? (Script: 4,481.)

Ordering: score relevance, then place the best items at the edges.

def score_relevance(query, docs):
    q = set(query.lower().split())
    return [len(q & set(d.lower().split())) / len(q) for d in docs]

def edges_first(items, scores):
    """Best item first, second best last, weakest in the middle."""
    ranked = [it for _, it in sorted(zip(scores, items), key=lambda p: -p[0])]
    return ranked[::2] + ranked[1::2][::-1]

Predict: with only two documents, does edges-first change anything?

History compaction keeps the last few turns verbatim and replaces the rest with a summary. Here the summary is a stub; in production it's a cheap model call ("summarize decisions, open questions, and user preferences").

def compact_history(turns, max_tokens, keep_recent=4, summarize=None):
    summarize = summarize or (lambda ts: "Summary of earlier turns: " + "; ".join(t[:40] for t in ts))
    total = sum(count_tokens(t) for t in turns)
    if total <= max_tokens:
        return turns
    old, recent = turns[:-keep_recent], turns[-keep_recent:]
    return [summarize(old)] + recent                     # a real system asks an LLM for the summary

Predict: if you re-summarize the summary every turn, what slowly happens to early details?

Tool selection keeps only the tools for the areas the query mentions. Real systems classify intent with a small model or embed tool descriptions and retrieve the closest ones.

def select_tools(query):
    hits = {area for area in set(TOOLS.values()) if area in query.lower() or (area == "files" and "file" in query)}
    return [t for t, area in TOOLS.items() if area in hits]

Predict: the query "search the git log and read the file that changed" selects which areas, and how many of the 30 tools?

Use the library

In production, each piece maps to a real component:

• Token counts: the provider's token-counting endpoint or the model's tokenizer (for example tiktoken for OpenAI models, or the Anthropic API's token-count endpoint).
• Retrieval and ordering: your RAG pipeline returns ranked chunks; apply a relevance threshold and a top-k, then order. A reranker gives better scores than word overlap.
• Compaction: a cheap model call that summarizes old turns, triggered by a token threshold. Some agent SDKs and APIs offer built-in context editing or compaction that clears old tool results automatically.
• Memory: a file the agent reads at session start (for example a project notes file), a key-value store of user preferences, or a vector index of past sessions queried like RAG.
• Tool selection: connect tools through MCP servers and expose only the relevant servers or tools per task.

The assembled context goes to the model as the system prompt, tool list, and messages of an ordinary API call.

Run it

cd guide
uv run code/28-prompting-and-context/context_engineering.py

Key output:

  total input                   12,200  (9.5% of the window)
  edges-first order (what the prompt sees, top to bottom):
    1.00  refund window is 30 days for damaged items
    0.20  refund goes to the original card
    ...
    0.40  damaged items ship back free
  14 turns, 1,120 tokens -> 5 messages, 416 tokens
  all tools 30 x 150 = 4,500 tokens; selected 10 -> 1,500 tokens (67% saved)
  naive (30 tools, 8 docs, full history)     call 1  9,200  call 20 13,950  total  231,500
  engineered (10 tools, 3 docs, compacted)   call 1  3,700  call 20  4,850  total   93,750

Full script: code/28-prompting-and-context/context_engineering.py.

5. Engineering

Layout for caching. Put stable content first and in a fixed order: system prompt, tool schemas, long reference documents that rarely change. Append new turns at the end; don't edit or reorder earlier messages. Prefix caches only match byte-identical prefixes, so a timestamp at the top of the system prompt can disable caching for every call.

Retrieval quality beats quantity. Three relevant chunks usually beat ten mediocre ones: fewer tokens, fewer distractors, less for the model to reconcile. Tune top-k and a relevance threshold on a labelled set, the same way you tune chunk size in RAG.

Repeat the critical instruction. For long inputs, put the key instruction both before the material and right before the question ("Using only the documents above, answer…"). Placing the question after long documents generally works better than before them.

Compaction loses information. A summary drops details, and summaries of summaries drift. Keep task-critical facts (IDs, decisions, constraints) in a structured notes block that is updated, not re-summarized. Let the agent re-fetch old tool outputs by ID rather than keeping them all.

Memory is untrusted input. Anything written to long-term memory from user or web content can carry injected instructions into future sessions. Store memories with provenance, let users view and delete them, and never treat a memory as a permission (see Securing AI systems).

Measure it. Log tokens per component per call. Build a small evaluation set where the answer depends on one fact placed at different depths in a long context, and check accuracy for your model before trusting long inputs.

Common mistakes

• Treating a large window as permission to include everything. You pay for every token on every call, and accuracy can drop.
• Forgetting the output reserve, so long inputs cut the answer short.
• Putting changing content (dates, user names) at the top of the system prompt, which defeats prefix caching.
• Exposing dozens of overlapping tools; the model picks the wrong one more often.
• Summarizing history with no record of key decisions, so the agent repeats work or contradicts itself.
• Storing everything in memory and retrieving it all; long-term memory needs the same relevance filtering as RAG.

6. Knowledge check

Practice exercise. A support bot has a 1,000-token system prompt, 2,000 tokens of tool schemas, and 3,000 tokens of retrieved documents per call, plus a 300-token query. Each turn adds 400 tokens of history. Compute total input tokens over 10 calls with full history, then with compaction that keeps the last 3 turns plus a 200-token summary.

Implementation challenge. Add a "needle" test to the script: build contexts of 20 documents where one contains a planted fact, place it at each position 1 to 20, and for each position record the fact's position as a fraction of the context. Then plug in a real model client and measure accuracy by position on 10 planted facts.

Summary

• The context window holds everything the model uses: system prompt, tools, examples, memory, retrieved documents, history, the query, and the output reserve.
• Long windows are available but not free: every token costs money and latency, and accuracy can drop on facts buried in long inputs.
• Select (fewer, more relevant items), order (stable prefix first, best items at the edges, question last), compress (summaries, trimmed tool output), and remember outside the window (retrieval over stored notes).
• Full history makes total input grow with T²; compaction makes per-call input constant after a few turns.
• Measure tokens per component and test long-context accuracy for your model instead of assuming it.

Next: The RAG pipeline

Related: RAG pipeline · Prompt engineering · Few-shot and chain-of-thought · LLM inference · Reliable agents

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Context Engineering: Windows, Budgets, Memory, and Retrieval.

Interview angle