Tokenization: How AI Reads Text

Tokenization: How AI Reads Text

AI models do not read text the way humans do. They cannot see letters and words as meaningful symbols. Instead, they convert text into numbers, and the first step in that process is called tokenization. Understanding tokens is essential to understanding how language models work and what their limitations are.

A token is a chunk of text. It can be a word, a part of a word, or even a single character. For example, the word understanding might be split into three tokens: under, stand, and ing. The word cat might be a single token. The model has a vocabulary, typically 32,000 to 128,000 tokens, that it knows how to process. Anything outside that vocabulary gets split into smaller pieces.

Tokenization affects everything about how a model performs. The number of tokens determines how much text the model can process at once, which is the context window. It also determines how fast the model generates output, because models generate one token at a time. A model that generates 100 tokens per second is producing about 75 words per second, which is faster than most people can read.

Different languages tokenize differently. English is very efficient because most models are trained primarily on English text. A word like the is a single token. But in other languages, the same meaning might require many more tokens. This means models are slower and more expensive to use for non English languages, and they may also perform worse because the tokenization breaks words in less natural ways.

Tokenization also creates some quirks. Models might struggle with simple tasks like counting letters in a word, because they do not see individual letters, they see tokens. A model might not know that strawberry has three r's because it sees strawberry as a single token. This is not a sign of stupidity, it is a fundamental limitation of how the model processes text.