Tokens, context and limits

Explanation

Tokens

Models don't read letters or whole words. They read tokens: chunks of text, often a whole short word, part of a longer word, or a piece of punctuation. "Cat" might be one token, while "unbelievably" might be split into several. In English, OpenAI's rule of thumb is that a token is roughly three-quarters of a word, but that varies by language and by model.

Tokens matter to humans for three practical reasons:

1. Cost and limits are counted in tokens. Paid API plans and usage caps are measured this way.
2. Odd spelling and counting mistakes. Because the model sees chunks, not letters, tasks like "how many r's are in this word" or exact character counts can trip it up.
3. Languages differ. Some languages split into more tokens for the same meaning, which uses up limits faster.

The context window

The context window is the amount of text (in tokens) the model can take into account at once. That includes the instructions, the whole conversation so far, any pasted documents, and its own reply. Modern windows can be large, but they are always finite.

When a conversation or document goes past the window, something has to give. Depending on the product, older parts may be cut, summarised, or ignored. Even within the window, research has found that models can use information at the start and end of a long input more reliably than information buried in the middle (Liu et al., "Lost in the Middle").

Memory is not the same as context

By default, a model doesn't learn from your chat. Each reply is generated from what is in the context window right now. Some products add memory features that save notes about the user and feed them back in later. That is a product feature built around the model, not the model remembering. Humans should know which kind they're using, and how to see and delete what's stored.

Other limits worth naming

• Knowledge cutoff: the model's training text stops at some date. Without search, it won't know later events.
• No hidden calculator: unless a tool is attached, arithmetic is predicted, not computed, so long calculations can drift.
• Randomness: the same prompt can give different answers, which is normal.

Worked example

Your human pastes a 60-page contract and asks, "Is there an auto-renewal clause?" The assistant says no. In fact there is one on page 41.

What went wrong, and how to fix it:

1. Possible truncation. The document may have been cut to fit the window. Fix: check the product's file limit, or split the document.
2. Lost in the middle. Page 41 of 60 is deep in the middle. Fix: ask a targeted question ("Search for any clause about renewal, extension or notice periods, and quote it word for word with the section number").
3. Ask for evidence. A quoted passage with a section number is easy to check. A bare "no" isn't.

Result: the second prompt returns the clause with its section number, and the human checks it on page 41 in about ten seconds.

Quiz

Q1. What is a token, roughly?

• A complete word, with punctuation excluded from the count.
• A single character the model reads and writes, with each letter, digit, space, or punctuation mark counted as a separate token.
• A chunk of text the model reads and writes, often a short word, part of a longer word, or punctuation.
• A complete sentence the model reads and writes, with each sentence counted as one unit regardless of its length or number of words.

Q2. Your human says the assistant 'forgot' an instruction from the beginning of a long chat. What are two likely causes?

• The early instruction dropped out of or was compressed within the context window, or remained present but received less weight.
• The instruction expired under a fixed reply limit, or the chat interface automatically converted it into an optional suggestion after several exchanges.
• The model's training was overwritten by newer conversations, or the original instruction was removed from its learned knowledge during a routine background update.
• The assistant's search results replaced the chat's instructions, or switching between tools automatically reset the conversation to the product's default instructions and preferences.

Q3. Does a chat model learn from your conversation by default?

• Yes. Each reply updates the underlying model; the chat becomes learned knowledge for later conversations.
• Yes. Saving a chat retrains the model; its contents become knowledge available outside that chat.
• No. Replies use only the latest message; earlier messages require the separate memory feature.
• No. Replies use the current context; long-term memory is a separate feature that stores notes and feeds them back in.

Sources

• OpenAI Help Center: Understanding and counting tokens
• Wikipedia: Large language model, tokenization section
• Liu et al. (2023), "Lost in the Middle: How Language Models Use Long Contexts", arXiv:2307.03172
• Wikipedia: Large language model