Animated explainer · 19 scenes · 4 min 44 s
A Transformer does not learn from rules someone wrote down. It learns from a single task, repeated over trillions of tokens of text: predicting the next token. This animation shows that cycle — from text to tokens, from attention to prediction, from error to weight adjustment — and what grows out of it: an internal representation of the world, from the next word to the next move.
In short
- A Transformer is trained on a single task: predicting the next token of a text.
- Text is split into tokens; each token becomes a vector (an embedding), and the attention mechanism decides which parts of the context matter for each token.
- During training, the prediction is compared with the true token; the error (the loss) is propagated backwards (backpropagation) and every weight is adjusted slightly.
- Repeated over trillions of tokens, this cycle gives rise to internal representations of concepts and relations — even of the state of a chess board.
- In use, the model no longer sees the answer: it writes text one token at a time, from the context.
The 19 scenes, one by one
Each scene shows the time it starts in the animation; press “Watch scene” to jump to it. The numbers the animation shows (probabilities, error, weights) are illustrative, not measurements from a real model.
-
Prologue ·
It all starts with text
Millions of sentences give the model one simple task: guess what comes next.
A language model is trained on a single task: given a piece of text, predict what comes next. There are no hand-written rules and no hidden objective — everything the model will ever be able to do grows out of this one task, repeated at enormous scale.
Watch scene -
01 · Data ·
One sentence at a time
The model is not given rules. It is given examples of language as it is actually used.
The training material is real text: books, articles, code, conversations. Every sentence is both question and answer — the first words are the context, and the next word is what the model has to get right.
Watch scene -
02 · Tokenization ·
Text becomes pieces
Words and word fragments are turned into tokens — small, numbered units.
Before any computation, text is split into tokens — whole words, pieces of words or punctuation marks — each mapped to a number in a fixed vocabulary, typically with tens of thousands of entries. Common words are usually a single token; rare ones are split into several. The numbers in the animation are illustrative.
Watch scene -
03 · Embeddings ·
Every token gets a position
A number becomes a point in a space with thousands of dimensions.
Each token is turned into a vector — a list of hundreds or thousands of numbers. Through training, tokens used in similar contexts end up with nearby vectors. Each token’s position in the sentence is encoded too, so that word order counts.
Watch scene -
04 · Attention ·
Every word looks at the others
Attention measures which parts of the context matter for understanding each token.
Attention is the heart of the Transformer: it lets each token gather information from the other tokens in the context, using learned weights. In “The cat sleeps in the sun”, this is how “sun” draws on “sleeps” and “cat”. In a model that generates text, each token can only look at the tokens that came before it.
Watch scene -
05 · Layers ·
The idea gets refined
Layer after layer, simple relations become increasingly abstract patterns.
A Transformer stacks dozens of identical blocks — attention followed by a small feed-forward network — and each block rewrites the representations of the one before. Early layers pick up local relations such as grammar; later ones, more abstract patterns.
Watch scene -
06 · Prediction ·
What will the next token be?
At the end, the model assigns a probability to every possible continuation.
At the top of the stack, the model scores every token in its vocabulary, and a function called softmax turns those scores into probabilities that add up to 100%. “sun” gets 62%, but “shade”, “box” and “garden” are possible continuations too.
Watch scene -
07 · Reality ·
The right answer was already known
During training, the prediction is compared with the token that actually came next.
During training the text already exists, so the true next token is known. Nobody has to label the data: the text itself provides the answer. This is called self-supervised learning.
Watch scene -
08 · Loss ·
Measuring the error
A single number sums up the distance between the model’s prediction and the correct continuation.
The gap between the predicted distribution and the right answer is summed up in one number, the loss — typically cross-entropy. The less probability the model gave to the correct token, the larger the error.
Watch scene -
09 · Backpropagation ·
The error travels backwards
The system works out how much each connection contributed to that error.
Backpropagation runs through the network from the end back to the start and computes, for each weight, the gradient: in which direction, and how strongly, nudging that weight would reduce the error.
Watch scene -
10 · Weights ·
Millions of tiny adjustments
Every weight changes a little, making the next prediction slightly better.
An optimizer — a variant of gradient descent such as Adam — changes each weight by a tiny amount, in the direction that lowers the error. A large model has millions, often billions, of weights, and every one of them is adjusted at each step.
Watch scene -
11 · Scale ·
Repeat. And repeat.
This cycle runs over huge amounts of text, many times over and in parallel.
The predict → measure → adjust cycle repeats over trillions of tokens, on thousands of processors working in parallel. So-called scaling laws show that the error falls predictably as data, model size and compute grow.
Watch scene -
12 · Learning ·
The error starts to fall
Without memorizing a list of rules, the model learns deep regularities.
The loss curve drops quickly at first, then ever more slowly. The model does not store a list of sentences: it compresses regularities — grammar, facts, styles — that let it predict well on text it has never seen.
Watch scene -
13 · Inside ·
An invisible map is born
Concepts, relations, styles and structures organize into internal representations.
Inside the network, concepts and relations organize into internal representations: directions and regions of the vector space that correspond to ideas such as “animal”, “motion” or “light”. Nobody programmed them; they emerge because they help with prediction.
Watch scene -
14 · World model ·
Predicting requires understanding structure
To continue a sentence well, the model has to capture something about how the world works.
To continue sentences about causes, times and distances well, it helps the model to represent those structures. How far this “understanding” goes is debated, but there is experimental evidence that models trained only to predict sequences form internal representations of the world.
Watch scene -
15 · New context ·
Now, without the answer
In real use, the model sees only the context and computes the next token.
In use — at inference time — there is no right answer to compare against and no weights to adjust. The model receives a new context, computes the distribution of the next token, picks one, appends it to the text and repeats: that is how an answer gets written, one token at a time.
Watch scene -
16 · Chess ·
A position is more than pieces
The model represents threats, space, king safety and possible plans all at once.
The same principle holds beyond language. Models trained only to predict the next move in games of Othello or chess develop internal representations of the board — where every piece is and, in chess, even an estimate of the players’ skill. The scene illustrates the idea: threats, center, king and space seen together.
Watch scene -
17 · Emergence ·
The move emerges from the state
“Knight to e5” does not come from an isolated rule, but from a rich representation of the position.
“Knight to e5” does not come from a rule of the form “if X happens, play Y”. It comes from a representation of the whole position, built layer by layer — the same mechanism that picks the next word of a sentence.
Watch scene -
Epilogue ·
Learning is adjusting predictions
From the next word to the next move: knowledge emerges from countless small corrections.
Context, representation, prediction. Everything a Transformer knows comes from countless small corrections to this cycle — from the next word to the next move.
Watch scene
Frequently asked questions
What is a Transformer?
A neural network architecture introduced in 2017 in the paper “Attention Is All You Need” (Vaswani et al.). It processes sequences with attention instead of recurrence, which makes it possible to train in parallel on huge amounts of data. It is the foundation of today’s large language models.
What is a token?
The unit text is split into before it enters the model: a word, a piece of a word or a punctuation mark. Each token maps to a number in a fixed vocabulary. Common words are usually one token; rare words are split into several.
What does the attention mechanism do?
For each token, it computes weights over the other tokens in the context and combines their information according to those weights. That is how a word like “it” can pick up which noun it refers to. In a model that generates text, each token only looks at the tokens before it.
What are the loss and backpropagation?
The loss is the number that measures how wrong a prediction was — usually cross-entropy, which grows when the model gives little probability to the right token. Backpropagation computes how much each weight contributed to that error, and the optimizer adjusts the weights in the direction that reduces it.
Does a model that only predicts the next word understand anything?
Predicting well forces a model to capture structure: grammar, facts, cause and effect. There is evidence that these models form internal representations of the world — for example, of the state of a game board. But this is not human understanding: models also make confident mistakes and invent plausible-sounding facts.
What is the difference between training and inference?
In training, the model sees the right answer, measures its error and adjusts its weights. At inference, the weights are frozen: the model only computes the next token from the context, appends it and repeats.
Further reading
The papers behind each idea in the animation:
- Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
- Sennrich, R., Haddow, B. & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016.
- Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533–536.
- Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015.
- Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv.
- Li, K. et al. (2023). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023.
- Karvonen, A. (2024). Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models. arXiv.