Context windows: what the model can see and why it forgets

Explanation

A reply starts with a supplied context

Think of a model call as a particular request to generate text from supplied material. For a text model, the context supplied for that call is its working memory: instructions, conversation excerpts, document passages and other text available to condition its reply. Generated tokens extend that working sequence as the answer unfolds.

This is different from knowledge encoded in learned parameters. Parameters reflect training and influence how the model interprets the input, but they are not a private transcript of earlier conversations. Ordinary generation does not update those parameters to store each new message. A previous decision can affect a later reply only if information about it reaches the model through the supplied context or another mechanism that makes it available.

The chat screen shows a conversation record maintained by an application. It does not necessarily show the exact request sent for the next reply. A file attachment may contribute selected extracts rather than every page. An earlier exchange may have been summarised. Something appearing in the interface is evidence that the application has a record of it, not proof that the current model call includes it verbatim.

How self-attention connects the pieces

In a common text-generating Transformer, causal self-attention lets each token position weigh information from earlier positions, as well as itself. “Causal” means that a position cannot look ahead at later tokens when predicting the continuation. During generation those future tokens have not been chosen yet.

A simple example is: “Maya put the blue folder beside the red notebook. Later, she opened the folder.” To process the later reference to “folder”, the model can draw on the earlier description. It does not have to squeeze every earlier detail through a single running sentence summary. Attention provides direct connections between positions in the available sequence.

The mechanism builds three numerical representations at each position, called a query, a key and a value. A query can be pictured as what this position is looking for. Keys provide features against which that query is compared. Values contain information to combine. These are learned numerical operations, not little written questions and answers inside the model.

Compatibility scores between a query and available keys become weights. The mechanism then forms a weighted mixture of the corresponding values. Several attention heads perform different such operations, and successive layers transform the resulting representations further. Position information helps the model distinguish where pieces occur; the input is not simply an unordered bag of words.

Full causal attention makes all earlier in-window positions eligible. Some architectures restrict or alter those connections for efficiency, so this description is a common mechanism rather than a promise about every model. Either way, no attention operation can recover an omitted passage merely because that passage still appears somewhere in the application's chat history.

Why length adds work

With ordinary full attention, each position compares against earlier positions. As the input grows, the number of position pairs grows roughly with the square of sequence length. Doubling the length therefore roughly quadruples these pairwise comparisons. This is a relationship in the mechanism, not a measured claim about any product's response time or price.

Efficient implementations and alternative architectures affect resource use. Pairwise comparison counts do not translate directly into a fixed increase in response time.

During generation, systems commonly cache intermediate key and value representations from earlier tokens. The cache avoids recomputing those representations from scratch after each new token. It still occupies resources, and attending over a longer retained sequence still requires work. A cache is a computational aid for the available sequence, not a durable notebook that automatically makes yesterday's chat available.

How an application creates conversational continuity

When you send a follow-up question, a chat application typically constructs a new request containing your latest message and relevant earlier conversation. The model's apparent memory across calls comes from that supplied material. The application may resend the complete history while it fits, maintain a compressed summary, or retrieve selected earlier passages.

Saved memory is another application mechanism. The app can store a preference or note outside the model and insert it into a later request. A stored note can influence a reply when supplied; storing it does not by itself teach the preference into the model's parameters. Saving, selecting and presenting information are separate steps.

Truncation removes information; compression changes it

When a request would exceed available capacity, the application must deal with that mismatch. Depending on its design, it may reject the request, remove older material, shorten documents, summarise history or retrieve only selected sections. There is no universal rule that the earliest message always disappears first.

Truncation means some material is cut away. If a relevant passage is missing from the actual input, the model cannot use that passage directly. Asking it to “look harder” does not restore the absent words. Supplying the excerpt or retrieving the source is the useful repair.

Compression can preserve the broad topic while losing a decisive qualification. A summary saying “Bring printed materials” may omit “except for the outdoor session”. Even a fluent summary can change the evidence available for later decisions. Preserve exact wording when exceptions, quantities, names or conditions matter.

Information can fit and still be missed

Research on long-context question answering and retrieval has found that performance can depend on where relevant information appears. In the experiments reported in “Lost in the Middle”, models often performed better when useful information appeared near the beginning or end than in the middle of a long input.

This is an empirical finding under particular tasks and conditions, not a law that every middle passage is ignored. Models, input structure and tasks differ. Do not turn the study into a fixed percentage or promise that moving text will solve every failure. The important distinction is between capacity, what can fit, and effective use, what the model successfully draws on.

For practical work, ask for a quotation and its source label, then compare them with the original. Put the operative question beside the evidence needed to answer it. For a large collection, inspect sections separately and keep an index of findings and locations. Treat these as ways to support review, not guarantees of completeness. Failure to find a passage is not sufficient evidence that the full source contains none.

Worked example

Your human is planning a community workshop. An early message says, “The main room seats thirty people; the side room seats twelve.” A later planning packet repeats several room descriptions. Buried among them is an exception: “For the evening session, the side room is unavailable.”

After a long discussion, the human asks whether a small evening group can use the side room. The assistant says yes. There are at least two plausible explanations: the exception was not supplied for that call, or it was supplied but not used correctly. Seeing the packet in the interface does not settle which explanation is right.

First, recover the original exception and give it a clear label, such as “Venue note, evening availability”. Supply the exact excerpt with the question: “For the evening session, can we use the side room? Quote the sentence that controls your answer.” The grounded answer should say no and quote the unavailability sentence. The seating capacity does not override that restriction.

Quiz

Q1. What does full causal self-attention allow at a token position?

• Access to every message the application has ever stored, including omitted chats.
• Access to future tokens that have not yet been generated.
• Weighted use of information from earlier positions and itself in the supplied sequence.
• Permanent storage of each earlier token in the model's learned parameters.

Q2. A model misses an exception in a long packet. What can you conclude from that failure alone?

• The application must have removed the exception before sending the request.
• The exception may be missing or may have been present but used poorly; the failure alone does not distinguish them.
• The context window must be too small to contain any part of the packet.
• The exception must be false because a model would use it if it were true.

Q3. Why does a longer sequence require more work with ordinary full attention?

• More positions produce more pairwise attention comparisons, growing roughly quadratically with length.
• Each extra token triggers a full retraining of the model on the conversation.
• Every extra token requires a search through all previous conversations.
• Length affects only storage; attention performs the same number of comparisons regardless of input.

Sources

• Vaswani et al. (2017), “Attention Is All You Need”
• Brown et al. (2020), “Language Models are Few-Shot Learners”
• Liu et al. (2024), “Lost in the Middle: How Language Models Use Long Contexts”
• Kwon et al. (2023), “Efficient Memory Management for Large Language Model Serving with PagedAttention”