Token
A token is the smallest unit of text a language model reads or writes, usually a word piece rather than a full word or single character.
Category
Industry concept
See all terms
What is a token?
A token is the smallest unit of text a language model reads or writes. Before a model does anything with a piece of text, a tokenizer breaks it into these units using a fixed vocabulary built from the model's training data, made up of common word pieces, whole words, characters, and punctuation. This solves a basic problem: a neural network operates on numbers, not language, so text has to become a sequence of numeric IDs before a model can process it.
How is a token different from a word?
A token and a word often get treated as the same thing, but they rarely match one to one. A common English word usually becomes a single token, while a rarer word, a name, a technical term, or a word in a less common language splits into several. This split-by-frequency design is what lets a model handle a word it has never seen: instead of failing on it, the tokenizer breaks it into smaller pieces it does recognize.
How does tokenization work?
A tokenizer builds its vocabulary by scanning huge amounts of training text and merging the most frequently paired characters into single units, repeating that process until it has a fixed set of tens of thousands of tokens. Reading new text means matching it against that vocabulary, picking the longest pieces that fit, and converting the result into numeric IDs the model can process. In English, one token works out to roughly four characters, so a document's token count depends on its vocabulary and structure, not its file size. Every LLM input and output gets measured and billed in tokens, increasing the need to provide models with knowledge targeted to the context.