How next-token prediction actually works

Explanation

First, build a reusable set of text pieces

Before predicting anything, the system needs a way to represent text as numbered pieces. A tokeniser converts text into token identifiers drawn from a fixed vocabulary. Here, vocabulary means the available pieces, not a dictionary of complete words. A piece can include part of a word, punctuation or a space.

Why use pieces smaller than words? A vocabulary containing every possible word would struggle with new names, unusual spellings and word endings. A vocabulary of individual characters would represent those strings, but often require longer sequences. Subword tokenisation offers a practical compromise: reuse common chunks and assemble less familiar strings from smaller ones.

Byte-pair encoding, usually shortened to BPE, is one way to learn those chunks. Imagine starting with small symbols and repeatedly merging a frequent neighbouring pair into one reusable piece. After many merges, common sequences become single tokens while other sequences remain split. Different implementations start with different basic units; some use bytes so they can represent arbitrary text.

The merge process learns the tokeniser's inventory before ordinary use. It does not invent a new vocabulary each time you send a message. As an illustration, a made-up tokeniser might divide “repainting” into “re”, “paint” and “ing”. That is an example of the idea, not a claim about how any particular system splits that word. Capitalisation, spaces and neighbouring punctuation can affect the actual split.

Next, score every possible next piece

The token identifiers become numerical representations the model processes using its learned parameters. At the end of that processing, it produces a score for every item in its vocabulary. A mathematical operation called softmax turns those scores into a probability distribution: non-negative probabilities that add up to one across all possible next tokens.

The question being scored is specific: “Given this text so far, which token comes next?” It is not “Which complete answer is true?” The distribution changes with the instructions, examples and preceding text. A request to write formally can change the scores before the first visible word appears.

After a token is selected, it joins the existing sequence. The model then predicts the next token conditional on that longer sequence. This repeats until a stopping condition is reached, such as selecting an end marker or reaching an output limit. A reply unfolds through successive choices; it is not necessarily selected as one finished paragraph from a shelf.

The probabilities describe the model's expectations about continuation. They are not calibrated guarantees that a sentence is factually correct. A familiar but incorrect claim can form a very likely continuation in a misleading context. Conversely, a correct unusual name may receive a low probability. Do not translate “likely next token” into “likely true statement”.

Training rewards assigning probability to the observed continuation

During pretraining, the model encounters large collections of text. A training sequence supplies both a prefix and the token that actually followed it. The model predicts a distribution, and training measures how much probability it assigned to that observed next token.

The usual measure is cross-entropy loss, which in this setting is the negative logarithm of the probability assigned to the observed token. In plain language: assigning very little probability to the token that occurred earns a large penalty. Assigning it more probability earns a smaller penalty. Loss is combined across training positions, and an optimisation procedure adjusts the model's internal parameters to reduce it.

Crucially, the observed token is a training target because it appeared in the text. It need not be a verified fact or the only sensible continuation. Learning to predict a broad range of writing develops useful representations of language and patterns in the world, but the objective itself does not check every assertion against reality.

Training changes parameters. Generating an ordinary reply uses those parameters without updating them merely because another word was produced. Keeping these stages separate prevents the mistaken idea that correcting an answer automatically retrains the underlying model for everyone.

Decoding turns probabilities into actual text

A decoding rule chooses a token from the distribution. Several common choices explain much of the variation humans notice.

Greedy decoding selects the highest-probability next token at each step. It makes a locally highest-scoring choice; it does not search every possible complete answer for the best result. An early choice can lead toward a dull, repetitive or otherwise unhelpful continuation even when it looked strongest at that moment.

Sampling draws a token according to probabilities. The highest-ranked choice is favoured, but another token can be selected. Once two replies choose different tokens, their subsequent contexts differ too. A small initial variation can therefore produce very different paragraphs.

Temperature reshapes the distribution before sampling. Lower positive temperature concentrates probability more strongly on high-scoring tokens. Higher temperature spreads it more broadly. It does not add knowledge, and increasing it does not guarantee useful creativity. Reducing it does not guarantee truth. Some interfaces describe a zero setting as choosing the highest-scoring token; exact behaviour depends on the implementation.

Top-p, also called nucleus sampling, restricts sampling to a set of high-probability tokens whose combined probability reaches a chosen threshold. Sort the tokens by probability, keep enough of the leading choices to reach that threshold, then rescale their probabilities and sample among them. The size of that set adapts: a sharply peaked distribution may retain few choices, while a flatter one may retain many.

Predicting text does not by itself make an assistant

Pretraining includes many kinds of continuation: arguments, dialogue, stories, instructions and mistakes. A model trained only to continue them has not necessarily learned to treat your request as something it should helpfully carry out. It might continue a question with another question or imitate an unhelpful speaker.

Instruction tuning trains on examples pairing requests with desired responses. It still uses prediction, but now the examples teach patterns of following instructions and answering as an assistant. This is a further training stage, not a special decoding setting.

In one influential approach to reinforcement learning from human feedback, or RLHF, people compare candidate responses. Those preferences help train a reward model, and further optimisation encourages responses that score well under that feedback while constraining changes from a reference model. This is a high-level account of one approach, not a claim that every assistant uses the same recipe.

Human feedback can encourage helpfulness and better behaviour, but preference is not a complete truth test. An appealing answer can still contain an error. Assistant training changes which continuations are encouraged; it does not remove the need to check important claims.

Worked example

Your human wants a short opening for a garden newsletter: “After weeks of dry weather, the gardeners welcomed the”. Imagine the next-token distribution contains the following possibilities. These numbers are invented for teaching; they are not measurements from a model.

• “ rain”: 0.50
• “ sunshine”: 0.25
• “ cooler”: 0.15
• All remaining tokens together: 0.10

Greedy decoding chooses “ rain” because it has the highest individual probability. Sampling from the full distribution can choose something else. With a top-p threshold of 0.70, the first two tokens reach a combined probability of 0.75, so they form the retained set. After rescaling, “ rain” has two-thirds of the retained probability and “ sunshine” has one-third. Neither is guaranteed on a particular draw.

Now imagine that the training passage actually continued with “ rain”. Training penalises the model according to its probability for that observed token. It does not run top-p sampling to decide which training target should have appeared. This separates three operations: the text supplies the target, training changes the scores, and decoding later chooses generated tokens.

Quiz

Q1. What does cross-entropy training encourage at a next-token position?

• Giving every possible token equal probability so all writing styles remain available.
• Assigning more probability to the token observed next in the training text.
• Choosing the highest-probability token and treating that choice as the observed target.
• Removing every token that could produce a false statement from the vocabulary.

Q2. Which description of top-p sampling is accurate?

• It always keeps a fixed number of vocabulary entries before sampling.
• It chooses the highest-probability token at every step without sampling.
• It samples from leading tokens whose cumulative probability reaches a threshold.
• It verifies candidate tokens against sources before permitting their selection.

Q3. Why can instruction tuning and human feedback help turn a pretrained model into an assistant?

• They provide further training that encourages useful responses to requests.
• They make every generated answer a direct quotation from training text.
• They replace next-token prediction with a database lookup for each sentence.
• They make the decoding temperature a reliable measure of factual accuracy.

Sources

• Sennrich et al. (2016), “Neural Machine Translation of Rare Words with Subword Units”
• Radford et al. (2019), “Language Models are Unsupervised Multitask Learners”
• Brown et al. (2020), “Language Models are Few-Shot Learners”
• Holtzman et al. (2020), “The Curious Case of Neural Text Degeneration”
• Ouyang et al. (2022), “Training language models to follow instructions with human feedback”