Context Size Explained

Context Size Explained

Context size, also called context window, is the maximum amount of text a model can process at once. It is one of the most important specifications of any language model, and it directly affects what you can do with the model.

Think of the context window as the model's working memory. It is everything the model can see when generating a response. If you give it a 100 page document and ask a question, the model can only answer if the document fits within its context window. If the document is too long, the model will forget parts of it or simply refuse to process it.

Context sizes have grown dramatically. Early models like GPT-3 had a context of only 2048 tokens, about 1500 words. Modern models handle 128K tokens, about 100,000 words. Some models like Claude 3 support 200K tokens, enough to process a book like The Great Gatsby in one go. There are even research models with 1 million token context windows.

There is a trade off with context size. Larger context windows require significantly more memory and compute. The attention mechanism, which we discussed earlier, scales quadratically with context length. Doubling the context quadruples the compute required. This is why very long context models are slower and more expensive to run.

For most practical purposes, 32K to 128K tokens is sufficient. This covers most documents, codebases, and conversations. Unless you are analyzing very long documents, extremely large context windows are not necessary. The key is to choose a model whose context window matches your use case, not the largest one available, to avoid unnecessary performance costs.