Topic modeling

1. Intuition

Two outputs. A topic model returns:

• Topics: each is a probability distribution over the vocabulary. A finance topic puts high probability on "stocks", "rates", "inflation".
• Document mixtures: each document is a distribution over topics. A news story about chip stocks might be 60% finance and 40% AI.

That second point is mixed membership: a document can belong to several topics at once, unlike k-means, where every point belongs to exactly one cluster.

Word co-occurrence is the only signal. LDA never sees meaning. It notices that "stocks" and "yields" keep showing up in the same documents, so they probably come from the same topic. That's why it needs many documents, and why it struggles with very short ones: a 6-word tweet carries little co-occurrence evidence.

Learning by repeated guessing. Collapsed Gibbs sampling starts by giving every word token a random topic. Then, one token at a time, it removes that token's assignment and re-picks a topic with probability proportional to two things: how much this document already uses each topic, and how much each topic already uses this word. Repeat for every token, hundreds of times. Rich-get-richer dynamics pull related words into the same topic and keep each document focused on a few topics.

The embedding alternative. BERTopic skips the word-count story: it embeds each document with a sentence encoder, clusters the embeddings, and describes each cluster by its most distinctive words. It captures meaning (synonyms land together) but gives each document one main topic by default.

Key distinction

A discovered topic is a statistical pattern in word use, not a real category. "Topic 7" might be a genuine theme, a writing style, a boilerplate signature, or a language. Name topics only after reading documents from them.

2. Visualization

Synthetic corpus: 60 documents of 30 words drawn from three hand-written word lists of 10 words each. Every fourth document mixes two topics 60/40. LDA recovers the three lists exactly and assigns the mixed documents two-topic mixtures. Topic numbers are arbitrary: LDA does not know which list is "finance".

Try it (predict first, then run the script and check):

1. Doc 3 was generated as 60% finance and 40% AI. Predict its inferred $\theta$ before looking at the output.
2. Set K=2 in the first gibbs_lda call. Predict which two true topics get merged, and whether topic diversity drops.
3. Raise alpha from 0.1 to 5. Predict what happens to the heatmap's pure documents.
4. Feed sklearn's LDA a TfidfVectorizer matrix instead of counts. Predict whether it errors, and whether the topics still make sense.

3. The math

Symbols

The generative story

Only the words $w$ are observed. Inference recovers $\theta$, $\phi$, and $z$. The model counts words, which is why LDA takes raw term counts (a bag of words), not TF-IDF weights.

Collapsed Gibbs update

"Collapsed" means $\theta$ and $\phi$ are integrated out, so we only sample the $z$'s. For one token with word $w$ in document $d$, with that token's own count removed (marked $\neg$):

After sampling, estimate

Worked example: one Gibbs update

$K = 2$ topics, $V = 30$, $\alpha = 0.1$, $\beta = 0.01$. The token is the word "model". With this token removed, its document has $n_{d0} = 7$ and $n_{d1} = 2$ tokens in each topic. Across the corpus "model" is assigned $n_{0,\text{model}} = 0$ times to topic 0 and $n_{1,\text{model}} = 9$ times to topic 1. Topic totals: $n_0 = 150$, $n_1 = 140$.

Denominators: $n_0 + V\beta = 150 + 30 \times 0.01 = 150.3$ and $n_1 + V\beta = 140 + 0.3 = 140.3$.

Normalize: the sum is $0.135333$, so $P(z = 0) = 0.000472 / 0.135333 = 0.003$ and $P(z = 1) = 0.997$.

The document leans toward topic 0 (7 vs 2 tokens), but topic 0 has never produced "model", so the word factor wins. This is how LDA separates words: a topic that never uses a word almost never gets it.

Choosing K

There is no single right $K$. Two cheap automatic signals:

• Topic diversity: the fraction of unique words across all topics' top-$n$ lists. 1.0 means no overlap; low values mean $K$ is too large and topics are splitting into near-duplicates.
• Topic coherence ($C_v$, NPMI): do a topic's top words co-occur in a reference corpus more than chance? Higher is better.

In the script, diversity of the top 6 words is 1.00 for $K = 2$ and $K = 3$, then 0.73 for $K = 5$ and 0.58 for $K = 8$. Note that $K = 2$ also scores 1.00 while merging two real topics: diversity catches too many topics, not too few. Read the topics.

4. Implementation

The full script is code/09-classical-nlp/topic_model_scratch.py (NumPy for the sampler, scikit-learn for the library comparison).

Build it from scratch

Step 1: map words to ids and start random. Every token gets a random topic. The three count tables are all the sampler needs: it never stores $\theta$ or $\phi$ while sampling.

def gibbs_lda(docs, K, alpha=0.1, beta=0.01, iters=200):
    vocab = sorted({w for d in docs for w in d})
    w2i = {w: i for i, w in enumerate(vocab)}
    V = len(vocab)
    W = [np.array([w2i[w] for w in d]) for d in docs]
    Z = [rng.integers(0, K, len(d)) for d in W]           # random initial topic per token
    ndk = np.zeros((len(W), K))                           # tokens in doc d assigned to topic k
    nkw = np.zeros((K, V))                                # times word w is assigned to topic k
    nk = np.zeros(K)                                      # tokens assigned to topic k

Predict: after initialization, roughly what fraction of a 30-token document's tokens is in each of 3 topics?

Step 2: fill the counts. One pass adds each token's random assignment to the three tables.

    for d, (ws, zs) in enumerate(zip(W, Z)):
        for w, z in zip(ws, zs):
            ndk[d, z] += 1
            nkw[z, w] += 1
            nk[z] += 1

Predict: what do ndk.sum(), nkw.sum(), and nk.sum() each equal?

Step 3: the sweep. For each token: subtract it from the counts (the $\neg$ in the formula), compute the $K$ unnormalized probabilities in one vectorized line, sample, and add it back under its new topic. Forgetting the subtraction counts the token as evidence for its own current topic.

    for _ in range(iters):
        for d, (ws, zs) in enumerate(zip(W, Z)):
            for i, (w, z) in enumerate(zip(ws, zs)):
                ndk[d, z] -= 1                            # remove this token's current assignment
                nkw[z, w] -= 1
                nk[z] -= 1
                p = (ndk[d] + alpha) * (nkw[:, w] + beta) / (nk + V * beta)   # [K]
                z = rng.choice(K, p=p / p.sum())          # resample its topic
                zs[i] = z
                ndk[d, z] += 1
                nkw[z, w] += 1
                nk[z] += 1

Predict: if beta were 0, what probability would a topic get for a word it has never been assigned?

Step 4: read off the estimates. Smoothed, normalized counts give $\theta$ and $\phi$.

    theta = (ndk + alpha) / (ndk.sum(1, keepdims=True) + K * alpha)        # [D, K] doc-topic
    phi = (nkw + beta) / (nkw.sum(1, keepdims=True) + V * beta)            # [K, V] topic-word
    return theta, phi, vocab

Predict: with $\alpha = 0.1$ and $K = 3$, what is the smallest possible $\theta_{dk}$ for a 30-token document?

Step 5: topic diversity.

def topic_diversity(tops):
    flat = [w for t in tops for w in t]
    return len(set(flat)) / len(flat)

Predict: 8 topics, 6 top words each, and 28 unique words among the 48. What is the diversity?

Use the library

scikit-learn's LDA uses online variational Bayes instead of Gibbs sampling; it is faster and deterministic for a fixed seed. Its priors are doc_topic_prior ($\alpha$) and topic_word_prior ($\beta$), both defaulting to $1/K$. NMF factorizes the count (or TF-IDF) matrix $X \approx WH$ with non-negative factors and is a fast, often sharper alternative.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation, NMF

cv = CountVectorizer(stop_words="english", min_df=2, max_df=0.9)   # raw counts; drop rare and ubiquitous words
X = cv.fit_transform(texts)                                          # [D, V] sparse
lda = LatentDirichletAllocation(n_components=10, random_state=0).fit(X)
doc_topic = lda.transform(X)                                         # [D, K], rows sum to 1
top = [cv.get_feature_names_out()[row.argsort()[::-1][:10]] for row in lda.components_]
nmf = NMF(n_components=10, init="nndsvd", random_state=0).fit(X)

For short or semantic text, BERTopic composes embeddings, dimensionality reduction, and clustering:

from bertopic import BERTopic   # pip install bertopic
topic_model = BERTopic(min_topic_size=15)                # default embeddings: all-MiniLM-L6-v2; UMAP; HDBSCAN
topics, probs = topic_model.fit_transform(texts)          # one topic id per doc; -1 = outliers
print(topic_model.get_topic_info().head())               # sizes and c-TF-IDF top words

Each cluster's words come from class-based TF-IDF: all documents in a cluster are concatenated into one, and a word scores $\text{tf}_{w,c} \cdot \log(1 + A / f_w)$, where $A$ is the average number of words per cluster and $f_w$ is the word's frequency over all clusters. min_topic_size defaults to 10. For mixtures, calculate_probabilities=True or approximate_distribution gives per-document topic distributions.

Run it

uv run code/09-classical-nlp/topic_model_scratch.py

Key lines of output:

unnormalised: topic0 = (7.1)(0.01)/150.3 = 0.000472, topic1 = (2.1)(9.01)/140.3 = 0.134861
normalised:   P(z=0) = 0.003, P(z=1) = 0.997

topic 0: researchers inference model attention training tokens
topic 1: congress governor vote tariffs president campaign
topic 2: market investors fed inflation stocks treasury
topic diversity (top 6): 1.00
doc 3: true mix [0.6 0.4 0. ], inferred theta [0.43 0.   0.56]
K = 5: diversity 0.73
K = 8: diversity 0.58

Doc 3 was 60% finance (topic 2 here) and 40% AI (topic 0), and LDA estimates 0.56 and 0.43. scikit-learn's LDA and NMF find the same three topics.

5. Engineering

When to pick what.

Documents longer than the embedding model's context window must be chunked (and the chunk topics aggregated) for BERTopic; LDA has no length limit.

Preprocessing decides quality. Remove stop words and boilerplate (signatures, disclaimers, templates), or they become topics. Filter rare words (min_df) and near-universal ones (max_df). Lemmatize if inflections split topics. Bigrams help ("interest rate").

Evaluation. Combine automatic coherence (gensim CoherenceModel with coherence="c_v"), topic diversity, and a human read of the top words and 10 to 20 documents per topic. Perplexity on held-out text often disagrees with human judgments of quality, so don't choose $K$ by perplexity alone. Check stability: rerun with different seeds and see whether topics persist.

Production. Topic ids change between training runs, so freeze a model and map ids to names before building dashboards. To track trends, assign new documents with transform and watch topic shares over time. For BERTopic, watch the -1 outlier share; if it is large, lower min_topic_size or use reduce_outliers.

Common mistakes

• Feeding TF-IDF into LDA. LDA's likelihood is over counts; use CountVectorizer.
• Naming topics from top words alone, then reporting them as categories. Read documents first.
• Picking $K$ from one metric. Diversity misses merged topics; coherence can favor generic topics.
• Leaving boilerplate in, then finding a "topic" that is an email signature.
• Running LDA on 10-word documents and blaming the algorithm.

6. Knowledge check

Practice exercise. $K = 2$, $V = 20$, $\alpha = 0.5$, $\beta = 0.1$. For a token of the word "vote", with the token removed: $n_{d0} = 3$, $n_{d1} = 3$; $n_{0,\text{vote}} = 1$, $n_{1,\text{vote}} = 5$; $n_0 = 98$, $n_1 = 98$. What is $P(z = 1)$?

Implementation challenge. Add a perplexity estimate and a simple coherence score (average PMI of top-word pairs, computed from document co-occurrence in the same corpus) to the script. Plot both against $K = 2..8$ and see whether they agree with topic diversity on the true $K = 3$.

Summary

• LDA models documents as mixtures of topics and topics as distributions over words; it outputs $\theta$ $[D, K]$ and $\phi$ $[K, V]$.
• Collapsed Gibbs sampling re-assigns each token with probability $\propto (n_{dk} + \alpha)(n_{kw} + \beta)/(n_k + V\beta)$.
• Small $\alpha$ gives focused documents, small $\beta$ gives focused topics; LDA needs raw counts.
• BERTopic (embeddings, UMAP, HDBSCAN, c-TF-IDF) wins on short semantic text; LDA wins on long mixed documents and low compute.
• Choose $K$ and judge quality with coherence, diversity, stability, and reading documents. Topics are patterns, not ground-truth categories.

Next: The neural-network forward pass

Related: Named entity recognition · From text to vectors · K-means · Density, hierarchical, and mixture clustering

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Topic Modeling — LDA and BERTopic.

Interview angle