Narrated animated explainer · 24 scenes · 13 min 16 s
A Transformer does not learn from rules someone wrote down. It learns from a single task, repeated over trillions of tokens of text: predicting the next token. This animation shows that cycle — from text to tokens, from multi-head attention to prediction, from error to weight adjustment — and answers the question that matters most: why such a simple objective produces models that understand, build a picture of the world and plan ahead, like a chess player who makes one move at a time with a strategy in mind.
In short
- A Transformer is trained on a single task: predicting the next token of a text.
- Text is split into tokens; each token becomes a vector (an embedding), and attention — with many heads in parallel, each following a different relation — decides which parts of the context matter for each token.
- During training, the prediction is compared with the true token; the error (the loss) is propagated backwards (backpropagation) and every weight is adjusted slightly, over trillions of tokens.
- Predicting well forces understanding: to guess the last word of a detective novel you have to have followed the plot. That is why internal representations of the world emerge — even of an Othello board the model never saw, which can be read and edited from inside.
- The model writes one token at a time, but plans ahead: it picks the rhyme before writing the line, and passes through “Texas” before answering “Austin”. Writing out the steps (a chain of thought) gives it even more room to think.
- Like a chess player: one move at a time, with a view of the board and a strategy. How far this goes is still an open question — models still make confident mistakes.
The 24 scenes, one by one
Each scene shows the time it starts in the animation; press “Watch scene” to jump to it. The longer text of each scene is exactly what the narration says, so it doubles as a transcript. The numbers the animation shows (probabilities, error, weights, moves) are illustrative, not measurements from a real model.
-
Prologue ·
It all starts with text
Millions of sentences give the model one simple task: guess what comes next.
A language model is trained on one single task: read a piece of text and guess what comes next. There are no hand-written rules and no hidden goal. It sounds almost trivial, like the autocomplete on a phone. The question this animation answers is how such a simple game produces a system that can explain, plan and reason. The short answer: to play this game well, a model has to build a picture of the world behind the words, and learn to think ahead.
Watch scene -
01 · Data ·
One sentence at a time
The model is not given rules. It is given examples of language as it is actually used.
The training material is real text: books, articles, code, conversations. Every sentence is both a question and its answer. The first words are the context, and the next word is what the model has to get right. Nobody explains grammar, physics or chess to it. Whatever it learns about them, it has to work out from the way people write about them.
Watch scene -
02 · Tokenization ·
Text becomes pieces
Words and word fragments are turned into tokens — small, numbered units.
Before any computation, the text is cut into tokens: whole words, pieces of words or punctuation marks, each with a number in a fixed vocabulary of tens of thousands of entries. Common words are usually a single token; rare ones are split into several. From here on, the model only ever sees sequences of numbers. The numbers on screen are illustrative.
Watch scene -
03 · Embeddings ·
Every token gets a position
A number becomes a point in a space with thousands of dimensions.
Each token becomes a vector: a list of hundreds or thousands of numbers, a point in a space with that many dimensions. Through training, tokens used in similar ways end up close together, and directions in that space come to mean things, such as singular and plural, or past and present. Each token’s position in the sentence is encoded too, so that word order counts.
Watch scene -
04 · Attention ·
Every word looks at the others
Attention measures which parts of the context matter for understanding each token.
Attention is the heart of the Transformer. Each token sends out a question, called a query. Every token before it offers a key, and where the query and a key match, that token’s information flows across. In “The cat sleeps in the sun”, this is how “sun” draws on “sleeps” and on “cat”. What to look at is not a fixed rule: it is learned, and it changes with the context.
Watch scene -
05 · Multi-head attention ·
Many ways of looking at once
Each head learns to follow a different relation: who does what, where, what came just before.
And attention does not happen just once. Every layer runs many attention heads in parallel, dozens in large models, each with its own queries, keys and values. One head learns to link a verb to its subject; another links a word to where the action happens; another simply looks at the word just before; another keeps track of the topic. Their results are then combined. It is like looking at a chess position with several pairs of eyes at once: material, threats, structure, the safety of the king. Nobody assigns these roles. They emerge because they help to predict.
Watch scene -
06 · Layers ·
The idea gets refined
Layer after layer, simple relations become increasingly abstract patterns.
A Transformer stacks dozens of these blocks: attention, followed by a small network that works on each token on its own. Each block reads what the previous ones left behind and writes new information on top, in a stream that runs through the whole model. Early layers pick up local patterns, like grammar. Deeper ones build meanings, relations and, as we will see, plans.
Watch scene -
07 · Prediction ·
What will the next token be?
At the end, the model assigns a probability to every possible continuation.
At the top of the stack, the model turns its final representation into a score for every token in its vocabulary, and a function called softmax turns those scores into probabilities that add up to one hundred percent. “Sun” gets sixty-two percent, but “shade”, “box” and “garden” are possible too. The model does not output a word. It outputs a whole landscape of possibilities.
Watch scene -
08 · Reality ·
The right answer was already known
During training, the prediction is compared with the token that actually came next.
During training, the text already exists, so the true next token is known. Nobody has to label anything: the text is its own answer key. This is called self-supervised learning, and it is why these models can learn from almost everything humanity has ever written.
Watch scene -
09 · Loss ·
Measuring the error
A single number sums up the distance between the model’s prediction and the correct continuation.
The gap between the prediction and the truth becomes a single number, the loss, usually the cross-entropy. If the model gave the right token a high probability, the loss is small. If it was confidently wrong, the loss is large. Everything the model will ever learn comes from trying to make this number smaller.
Watch scene -
10 · Backpropagation ·
The error travels backwards
The system works out how much each connection contributed to that error.
Backpropagation sends the error back through the network, from the last layer to the first, and works out, for each of billions of weights, in which direction, and by how much, a small change would have reduced the error. It is the chain rule from calculus, applied on an enormous scale.
Watch scene -
11 · Weights ·
Millions of tiny adjustments
Every weight changes a little, making the next prediction slightly better.
An optimizer, such as Adam, then nudges every weight by a tiny amount in the direction that lowers the error. No single step teaches anything you could point to. Knowledge accumulates the way a riverbed forms: from countless small corrections, all pushing the same way.
Watch scene -
12 · Scale ·
Repeat. And repeat.
This cycle runs over huge amounts of text, many times over and in parallel.
Predict, measure, adjust. The cycle repeats over trillions of tokens, on thousands of chips working in parallel for months. So-called scaling laws show that the error falls predictably as data, model size and compute grow. And with scale, abilities keep appearing that nobody trained for directly.
Watch scene -
13 · Learning ·
The error starts to fall
Without memorizing a list of rules, the model learns deep regularities.
The loss falls fast at first, then more and more slowly. The easy patterns, like spelling, grammar and common phrases, are learned early. What remains is harder: facts, logic, people’s intentions, how situations unfold. Those last fractions of error are where the understanding lives.
Watch scene -
14 · Why it’s hard ·
To predict, you have to understand
To guess the last word of a detective novel, you have to have followed the whole plot.
This is why predicting the next token is not a shallow trick. Picture a detective novel that ends: “and the murderer was…”. To predict that one word, it is not enough to know which names are common. You have to track the characters, the clues, the alibis and the motives across hundreds of pages. It is an example Ilya Sutskever, one of the founders of OpenAI, likes to use. The same holds for a proof, a program or a legal argument. The better the prediction, the more of the world behind the text the model has to capture. Prediction is the test; understanding is what it takes to pass it.
Watch scene -
15 · Inside ·
An invisible map is born
Concepts, relations, styles and structures organize into internal representations.
Inside the network, concepts and relations organize themselves into internal representations: directions and regions in the space of vectors that correspond to ideas like “animal”, “motion” or “light”. Researchers can find them, and even switch them on and off to change what the model says. Nobody programmed them. They emerged because they help to predict.
Watch scene -
16 · World model ·
Predicting requires understanding structure
To continue a sentence well, the model has to capture something about how the world works.
Some of these representations look like a model of the world. Researchers found that language models encode the latitude and longitude of places, and when historical figures lived, as straight lines inside their vector space, learned from nothing but text. To continue sentences about causes, times and distances well, it pays to represent causes, times and distances.
Watch scene -
17 · A board inside ·
A board nobody showed it
Trained only on lists of moves, a model builds its own picture of the board.
The clearest evidence comes from games. Researchers trained a Transformer only on lists of Othello moves, like “d3, c5, f6”: no pictures of the board, no rules. Then they looked inside. The state of the whole board, which disc sat on which square, could be read straight out of its activations. And when they edited that internal board, the model’s next moves changed to fit the edited position. It had not memorized sequences. It had built the board. A model trained the same way on chess games does the same, and even tracks how strong the players seem to be.
Watch scene -
18 · Planning ·
One word at a time, with a plan
Before writing a line of verse, the model has already chosen the word it will rhyme with.
Writing one token at a time does not mean thinking one token at a time. When a model writes a rhyming couplet, researchers found that it picks the rhyming word before it starts the second line, and then builds the line to land on it. Change that internal choice, and the whole line changes. Asked for the capital of the state where Dallas is, it first represents “Texas” internally, and only then says “Austin”. Like a chess player, it makes one move at a time, with a plan already in mind.
Watch scene -
19 · New context ·
Now, without the answer
In real use, the model sees only the context and computes the next token.
In use, at inference, there is no right answer to compare with and no weights to adjust. The model receives a new context, computes the distribution for the next token, picks one, adds it to the text, and repeats. Every word of an answer is a new prediction, made with everything written so far in view.
Watch scene -
20 · Reasoning ·
Thinking out loud
Every token it writes becomes context for the next: writing out the steps gives it room to reason.
There is one more piece. A single pass through the network has a fixed amount of computation. But every token the model writes goes back in as context. So when it writes out the intermediate steps, a chain of thought, it gives itself a scratchpad and more computation, the way a chess player calculates variations before touching a piece. Today’s reasoning models are trained, with reinforcement learning, to think this way for longer before answering, and they solve much harder problems as a result.
Watch scene -
21 · Chess ·
A position is more than pieces
The model represents threats, space, king safety and possible plans all at once.
Back to chess. A strong player does not see thirty-two pieces. They see threats, weak squares, an exposed king, a plan. Language models trained only to predict the next move in chess notation develop internal representations of the same kind: where every piece is, and even an estimate of the players’ skill. And a Transformer trained to predict strong moves, with no search at all, reached grandmaster level in blitz.
Watch scene -
22 · Emergence ·
The move emerges from the state
“Knight to e5” does not come from an isolated rule, but from a rich representation of the position.
So “knight to e5” does not come from a rule like “if this, play that”. It comes from a picture of the whole position, built layer by layer, and from where that position can lead. The same machinery chooses the next word of a sentence: one move at a time, each move informed by a view of the whole board.
Watch scene -
Epilogue ·
Learning is adjusting predictions
From the next word to the next move: knowledge emerges from countless small corrections.
Context, representation, prediction. Everything a Transformer knows comes from countless small corrections to that cycle. Predicting the next token is the objective, not the limit: to predict well, a model has to represent the world behind the words and plan where they are going. How far this goes is still an open question. These models still make confident mistakes, and their picture of the world can be patchy. But the next word is only the visible move. The strategy lies underneath.
Watch scene
Frequently asked questions
What is a Transformer?
A neural network architecture introduced in 2017 in the paper “Attention Is All You Need” (Vaswani et al.). It processes sequences with attention instead of recurrence, which makes it possible to train in parallel on huge amounts of data. It is the foundation of today’s large language models.
What is a token?
The unit text is split into before it enters the model: a word, a piece of a word or a punctuation mark. Each token maps to a number in a fixed vocabulary. Common words are usually one token; rare words are split into several.
What does the attention mechanism do?
For each token, it compares that token’s query with the keys of the tokens before it and combines their information (the values) according to how well they match. That is how a word like “it” can pick up which noun it refers to. In a model that generates text, each token only looks at the tokens before it.
What is multi-head attention?
In every layer, attention runs several times in parallel — dozens of “heads” in large models — each with its own queries, keys and values. Each head learns to follow a different relation: the subject of a verb, the previous word, what a pronoun refers to, the topic. The results of all of them are then combined. Nobody assigns the roles: the heads specialize during training, because that helps prediction.
What are the loss and backpropagation?
The loss is the number that measures how wrong a prediction was — usually cross-entropy, which grows when the model gives little probability to the right token. Backpropagation computes how much each weight contributed to that error, and the optimizer adjusts the weights in the direction that reduces it.
How can predicting the next word lead to reasoning?
Because predicting well is hard: to guess the murderer’s name on the last page of a detective novel, you have to have followed the whole plot. To bring its error down, a model has to represent what lies behind the text. There is experimental evidence for this: a Transformer trained only on Othello moves forms an internal picture of the board that can be read and edited from inside; and in language models, researchers see the model pick the rhyme before writing the line, and pass through “Texas” before answering “Austin”. Like a chess player, it plays one move at a time, but each move comes from a view of the board and a plan.
So do these models understand the way we do?
Not in the same way, and how far their understanding goes is still an open question. Their internal representations can be partial or incoherent, and models still make confident mistakes and invent plausible-sounding facts. What the evidence shows is that predicting the next token is not just surface statistics: it forces the model to build structure.
What is the difference between training and inference?
In training, the model sees the right answer, measures its error and adjusts its weights. At inference, the weights are frozen: the model only computes the next token from the context, appends it and repeats. When it writes out intermediate steps before answering (a chain of thought), each step goes back in as context and gives it more room to think.
Further reading
The papers behind each idea in the animation:
- Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017.
- Sennrich, R., Haddow, B. & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. ACL 2016.
- Clark, K., Khandelwal, U., Levy, O. & Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT’s Attention. BlackboxNLP 2019.
- Voita, E. et al. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019.
- Rumelhart, D. E., Hinton, G. E. & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533–536.
- Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015.
- Kaplan, J. et al. (2020). Scaling Laws for Neural Language Models. arXiv.
- Gurnee, W. & Tegmark, M. (2024). Language Models Represent Space and Time. ICLR 2024.
- Li, K. et al. (2023). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023.
- Nanda, N., Lee, A. & Wattenberg, M. (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. BlackboxNLP 2023.
- Karvonen, A. (2024). Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models. arXiv.
- Lindsey, J. et al. (2025). On the Biology of a Large Language Model. Transformer Circuits.
- Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
- DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.
- Ruoss, A. et al. (2024). Grandmaster-Level Chess Without Search. arXiv.