Skip to content
ML to LLMs
Labs / LLM and RAG Lab

Tokenization explorer

Type text, step through BPE merges, and watch tokens fill a context window.

Theory and worked examples: Tokenization and pretraining

Tokenization explorer

A byte-pair encoder trained in your browser on a six-sentence ML text (127 merges learned). Real tokenizers learn 50k–200k merges from huge corpora.

Merge 40 of 127: "▁" + "n" → "▁n" doesn't appear in your text, so nothing changes

▁the▁transformer▁model▁predicts▁tokens.▁unbelievable▁hyperparameters!

Each coloured group is one word; each box inside it is one token. ▁ marks the start of a word (the space before it).

Context window

system prompt your text2 tokens left for the answer
Where the merges come from: the training text, and the merges that change your text

The tokenizer counted every adjacent pair of symbols in this text (repeated 3 times) and merged the most frequent pair, 127 times.

the model learns to predict the next token from the previous tokens
training the model on more data makes the predictions better
a transformer model uses attention to weigh the tokens in its context
the learning rate controls how fast the model learns during training
tokenization splits text into tokens and the model predicts tokens
the model is trained to predict the next token in the training data

Merges that change your text (32 of 127). Click one to jump to it:

Words common in the training text ("the", "model", "token") become single tokens; rare or unseen words stay in pieces. Most merges never touch a given sentence, which is why the count only drops at some steps.