Tokens and context windows

A large language model (LLM) is a program trained on a huge amount of text to do one thing: given some text, predict what comes next. It writes one small piece, adds it to the text, and predicts again, until it decides to stop or hits a limit. Answering questions, writing code and calling tools are all built on that loop.

The small pieces are tokens, not words or letters. A tokenizer splits text into chunks it has seen often: a common word is usually one token, a rare word is split into several, and spaces and punctuation count too. Each model family has its own tokenizer, so the same text can be a different number of tokens on different models.

For English, a rough rule is about four characters, or three quarters of a word, per token. Code, numbers and many other languages use more tokens for the same length of text. When the count matters (for cost or limits), use the model's own token counter rather than the rule.

text = "Retrieval adds the right documents to the prompt."
print(len(text.split()), "words,", len(text), "characters")
print("roughly", round(len(text) / 4), "tokens")

The context window is the most tokens a model can handle in one request: your instructions, the conversation so far, any documents you include, and the answer it writes. If it doesn't fit, something has to give: the app drops or summarises older messages, or the request is refused. Windows have grown from a few thousand tokens to hundreds of thousands in many current models; check the figure for the model you use.

The model keeps no memory between requests. A chat app sends the whole conversation again on every turn, which is why long chats get slower and cost more. Anything the model should know, like today's date, the customer's order or the document in question, has to be in the context. What it learned in training is fixed, and stops at its training cut-off date.

A long context isn't free even when it fits. You pay for every token, answers start more slowly, and models can overlook details buried in the middle of a very long context. Put in what the task needs, and put key instructions where they're easy to find.