The RAG pipeline

1. Intuition

1. Ingest: load documents (PDFs, HTML, tickets), extract clean text, keep metadata (source, date, access permissions).
2. Chunk: split documents into retrievable pieces. Too small and a chunk lacks the context to answer; too large and the relevant sentence is diluted (and costs more tokens).
3. Embed and index: turn each chunk into a vector (dense retrieval) and/or a term index (sparse retrieval), stored with metadata.
4. Retrieve: turn the user's query into the same representation and fetch the top-$k$ chunks, optionally filtered by metadata (only this user's documents, only this product).
5. Rerank: a slower, more accurate model (often a cross-encoder) rescores the candidates.
6. Construct context: place the best chunks in the prompt with source labels.
7. Generate: instruct the model to answer only from the provided context, cite sources, and say when the answer isn't there.
8. Evaluate: measure retrieval and answers separately (next lesson).

2. Visualization

Try it (predict first, then check):

1. Pick the Retrieval miss question. Predict whether a better prompt could fix the answer, then turn on semantic matching instead.
2. Shrink chunks to 8 words. Predict what happens to evidence recall, then raise the chunk size and compare.
3. Set k = 1 and try each question in turn. Which still get their evidence into the context, and does the reranker change any of them?
4. Keep the default question and move the chunk size from 20 to 30 words. Predict where the evidence ranks, then check the worked example below to see why it moves.

3. The math

Symbols

Dense retrieval

Hybrid scoring (one common form)

where $\tilde{s}$ are normalized scores. Reciprocal rank fusion, which adds $1/(60 + \text{rank})$ from each retriever, is a popular score-free alternative.

Chunking arithmetic

With chunk size $S$ words and overlap $O$, consecutive chunks start every $S - O$ words. A document of $N$ words produces about $\lceil (N - O)/(S - O) \rceil$ chunks. Overlap reduces the chance that an answer sentence is cut in half, at the cost of more chunks.

Worked example: one query, step by step

Collection: four short synthetic policy documents. Query: "Are shipping costs refunded if my item was broken on arrival?" Scores use the course's TF-IDF from text to vectors: raw counts times $\ln(N/\text{DF})$, common words removed, cosine similarity. These are the numbers the lab shows.

With 20-word chunks and 5-word overlap (10 chunks):

Now change only the chunk size, to 30 words (7 chunks):

Same documents, same question, and the evidence drops from rank 1 to rank 2. The bigger refunds chunk also carries sentences about digital downloads and payment methods, which dilute its match, while the shipping chunk says "shipping" three times. With $k = 3$ the model still sees the evidence; with $k = 1$ it would not, and it would likely answer from shipping times. A reranker that reads the query and chunk together moves the refunds chunk back to the top. Chunk size is a retrieval hyperparameter: tune it against a labeled set, not by eye.

If you reproduce this with scikit-learn's TfidfVectorizer, the numbers differ, because its default smooths IDF to $\ln\frac{1+N}{1+\text{DF}} + 1$; with 20-word chunks it happens to rank the shipping chunk first (0.437 vs 0.425). Either way the point stands: lexical scores are fragile, so measure retrieval.

Worked example: a retrieval miss

Query: "Can I get reimbursed for a game I never opened?" The answer exists ("Digital downloads are refundable only if they have not been accessed"), but the query shares no words with it: reimbursed versus refundable, game versus digital downloads, never opened versus not accessed. Every TF-IDF score is 0. A dense embedding retriever would likely match the meaning; a hybrid retriever gets the best of both. Without a fix, the generator either says "I don't know" (good) or invents an answer (bad).

4. Implementation

The full script is code/16-rag/rag_scratch.py (NumPy only, fully offline). It uses the lab's four synthetic documents, stop words, and scoring, so it prints the worked example's numbers. Two parts are deliberately fake so nothing needs a download: query rewriting uses a hand-written synonym map instead of an LLM, and the reranker is a scoring rule instead of a trained cross-encoder. The pipeline around them is the real shape.

Build it from scratch

Step 1: chunk. Fixed windows are the baseline: $S$ words, starting every $S - O$ words. Sentence chunking packs whole sentences up to a word budget, so an answer sentence is never cut in half. Each chunk keeps its source, which you need for citations and filters later.

def fixed_chunks(size, overlap):
    out, step = [], max(1, size - overlap)
    for doc, text in DOCS.items():
        w = text.split()
        out += [(doc, " ".join(w[i:i + size])) for i in range(0, max(1, len(w) - overlap), step)]
    return out                                    # [(source, text)]

def sentence_chunks(max_words):
    out = []
    for doc, text in DOCS.items():
        cur = []
        for sent in re.split(r"(?<=\.)\s+", text):  # never cut a sentence in half
            if cur and len(" ".join(cur + [sent]).split()) > max_words:
                out.append((doc, " ".join(cur))); cur = []
            cur.append(sent)
        out.append((doc, " ".join(cur)))
    return out

Predict: the refunds document has 45 words. With $S = 20$ and $O = 5$, how many fixed chunks does it produce? (Use the chunking formula in section 3.)

Step 2: index and retrieve. Every chunk becomes a TF-IDF vector (raw count times $\ln(N/\text{DF})$), stacked into a matrix M of shape [n_chunks, |vocab|]. Retrieval is one matrix-vector product and a sort by cosine. A dense retriever swaps vec for an embedding model; the rest is unchanged.

def retrieve(chunks, index, query_terms, k):
    vec, M = index
    q = vec(query_terms)
    s = M @ q / (np.linalg.norm(M, axis=1) * np.linalg.norm(q) + 1e-12)
    order = sorted((i for i in range(len(chunks)) if s[i] > 0), key=lambda i: (-s[i], i))
    return [(i, float(s[i])) for i in order[:k]]

Predict: the miss query ("reimbursed ... game ... never opened") shares no terms with any chunk. What does retrieve return for it?

Step 3: rewrite the query. When the user's words differ from the document's words, add the document's words. Production systems ask an LLM for paraphrases (multi-query), for a hypothetical answer to embed instead of the question (HyDE), or for sub-questions (decomposition). Here a synonym map plays the LLM.

def rewrite(query):
    return [x for t in terms(query) for x in [t] + SYNONYMS.get(t, [])]   # an LLM rewriter in production

Predict: rewriting adds "refund" to any query containing "back". What could go wrong for "How do I go back to the previous page?"

Step 4: rerank. A first-stage retriever scores the query and each chunk separately. A reranker reads them together and can reward query words that appear close together in the chunk. A real cross-encoder is a transformer over the pair [query] [SEP] [chunk]; this rule (coverage of query terms times their density) is the same rule the lab's reranker uses.

def rerank_score(chunk_text, query):
    toks = terms(chunk_text)
    q = set(rewrite(query))
    pos = [i for i, t in enumerate(toks) if t in q]
    if not pos:
        return 0.0
    coverage = len({toks[i] for i in pos}) / max(1, len(terms(query)))
    span = pos[-1] - pos[0] + 1 if len(pos) > 1 else len(toks)
    return round(coverage * (1 + len(pos) / span), 4)

Predict: a cross-encoder is far more accurate than a bi-encoder. Why not use it for first-stage retrieval over a million chunks?

Step 5: build a grounded prompt and check citations. Number the sources, tell the model to answer only from them and to say when they don't contain the answer. After generation, check that every cited number points at a real source; a citation to [4] when you sent two sources is a hallucinated citation.

def build_prompt(question, chunks, hits):
    sources = "\n".join(f"[{n}] ({chunks[i][0]}) {chunks[i][1]}" for n, (i, _) in enumerate(hits, 1))
    return ("Answer using ONLY the sources below. Cite sources like [1]. "
            "If the sources do not contain the answer, say you don't know.\n\n"
            f"Sources:\n{sources}\n\nQuestion: {question}")

def check_citations(answer, n_sources):
    cited = {int(c) for c in re.findall(r"\[(\d+)\]", answer)}
    return {"cited": sorted(cited), "invalid": sorted(c for c in cited if not 1 <= c <= n_sources), "uncited": not cited}

Predict: the citation check passes. Does that prove the answer is supported by source [1]?

Use the library

The same pipeline with a real embedding model:

# Index (once): chunk, embed, store with metadata
from sentence_transformers import SentenceTransformer          # pip install sentence-transformers
import numpy as np

encoder = SentenceTransformer("all-MiniLM-L6-v2")
chunks = [c for doc in docs for c in chunk(doc.text, size=200, overlap=40)]
C = encoder.encode([c.text for c in chunks], normalize_embeddings=True)    # [n_chunks, d]

# Query time: retrieve, (rerank), build a grounded prompt
def answer(question, k=5):
    q = encoder.encode([question], normalize_embeddings=True)[0]
    top = np.argsort(-(C @ q))[:k]                                          # cosine = dot for normalized vectors
    context = "\n\n".join(f"[{i}] ({chunks[i].source}) {chunks[i].text}" for i in top)
    prompt = (
        "Answer using ONLY the sources below. Cite sources like [3]. "
        "If the sources do not contain the answer, say you don't know.\n\n"
        f"Sources:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt), top

What production adds: the in-memory array becomes a vector database or a search engine with vector support (vector search), with metadata filters for permissions and freshness. Step 4 becomes a trained cross-encoder (sentence_transformers.CrossEncoder or a hosted rerank API) over the top 20 to 100 candidates. Step 3 becomes an LLM call, and step 1 becomes a splitter that respects headings, paragraphs, and code blocks.

Run it

uv run code/16-rag/rag_scratch.py

Key lines of output (synthetic documents):

10 chunks of 20 words: [('refunds', 0.408, True), ('shipping', 0.347, False), ('shipping', 0.112, False)]
7 chunks of 30 words: [('shipping', 0.303, False), ('refunds', 0.283, True)]
30-word chunks after rerank: [('refunds', 1.0476), ('shipping', 0.4167)]
terms: ['get', 'reimbursed', 'game', 'never', 'opened'] -> hits: []
-> top hit: [('refunds', 0.517, True)]
fixed 20w/5 overlap : 10 chunks, recall@1 0.4, with rewrite 1.0
sentences <= 20w    : 11 chunks, recall@1 0.6, with rewrite 1.0
citation check on 'Yes, if it arrived damaged [1][4].': {'cited': [1, 4], 'invalid': [4], 'uncited': False}

The first two lines are the worked example's tables. The reranker puts the refunds chunk back on top at 30 words, and rewriting fixes the miss. Read the recall lines with care: five questions means each one moves recall by 0.2, and the synonym map was written for these exact questions, so "1.0 with rewrite" shows the mechanism, not a result to expect on real data. The ranking metrics and the chunk-size figure are in code/16-rag/rag_eval.py.

5. Engineering

Retrieval first. Measure retrieval recall@k on a labeled set of (question, relevant chunk) pairs before tuning prompts. If the evidence isn't retrieved, no prompt can fix the answer.

Chunking. Split on structure (headings, paragraphs) rather than fixed word counts where possible; keep titles and section headers with chunks; use overlap; tune size on your retrieval metrics.

Hybrid search and reranking. Sparse retrieval handles exact terms, IDs, and rare words; dense handles paraphrases. Hybrid is a strong default. Rerank the top 20 to 100 candidates with a cross-encoder, accepting the extra latency.

Query rewriting. Multi-query (several paraphrases, results merged), HyDE (embed a model-written hypothetical answer instead of the question), and decomposition (split a two-part question into two searches) all fix vocabulary mismatch and multi-hop questions, at the cost of an extra LLM call before retrieval. Rewrites can also drift from what the user asked, so keep the original query in the mix and measure recall with and without the rewriter.

Metadata filtering and permissions. Filter by access rights before retrieval results reach the model. Never rely on the model to withhold documents the user shouldn't see.

Freshness and versioning. Re-index on document changes; version the index together with the embedding model; record which index version answered each request.

RAG versus fine-tuning. RAG changes the runtime context, so updating knowledge means re-indexing, and answers can cite sources. Fine-tuning changes weights and is the right tool for behavior and format, not for facts that change.

Watch out

Failure modes. Parsing failures (tables and PDFs turned into garbage); chunks that split the answer; vocabulary mismatch for sparse retrieval and exact-term misses for dense retrieval; stale indexes; too much context, which dilutes attention and costs money; the model ignoring context and answering from memory; retrieved documents containing prompt injections (see security).

Common mistakes

• Judging a RAG system by how fluent its answers sound.
• Using different embedding models (or versions) for documents and queries.
• Skipping the "I don't know" instruction and the evaluation that rewards it.
• Retrieving across all users' documents and filtering afterwards.

6. Knowledge check

Practice exercise. A 1,000-word document is split into chunks of 200 words with 50 words of overlap. How many chunks result, and how many words are stored in total?

Implementation challenge. Extend the script: add a dense retriever (any sentence-embedding model), combine it with TF-IDF using reciprocal rank fusion, and report evidence recall@3 for sparse, dense, and hybrid on 20 questions you write about the documents, including paraphrased ones.

Summary

• RAG retrieves relevant chunks at query time and asks the model to answer from them with citations.
• Chunk size and overlap trade answer completeness against context length and precision.
• Sparse, dense, and hybrid retrieval have complementary failure modes; reranking improves precision.
• Most failures are retrieval failures: measure recall@k first.
• RAG changes context, not weights, which makes it the right tool for private and changing knowledge.

Next: Vector search

Related: From text to vectors · Adapting LLMs · Model card: embedding model · Model card: reranker

Go deeper, code-first (AI Engineering from Scratch): Text Summarization · Question Answering Systems · Relation Extraction & Knowledge Graph Construction · Document and Diagram Understanding · ColPali and Vision-Native Document RAG · Multimodal RAG and Cross-Modal Retrieval

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): RAG (Retrieval-Augmented Generation), Advanced RAG (Chunking, Reranking, Hybrid Search), Chunking Strategies for RAG, Chunking Strategies, Compared, Cross-Encoder Reranker, Query Rewriting: HyDE, Multi-Query, and Decomposition.

Interview angle