Transformer
A transformer is the neural network architecture behind modern language models, built on self-attention.
Category
Industry concept
See all terms
What is a transformer?
A transformer is a neural network architecture that processes an entire sequence of tokens at once instead of one token at a time. Every large language model in use today is built on this architecture.
Before transformers, models read text sequentially, one token after another, carrying a running summary forward as they went. That summary got compressed with every step, so information from early in a long passage kept getting diluted or lost over time. Transformers process the whole sequence in parallel and let every token look directly at every other token, so a fact at the start of a document has the same direct path to the model's attention as one at the end.
How does a transformer differ from self-attention?
A transformer is different from self-attention: self-attention is the single mechanism that lets tokens compare themselves against each other, while a transformer is the full architecture built around that mechanism. A transformer stacks self-attention with feedforward layers, positional encoding, and normalization, repeated across many layers. Self-attention is the part that decides what a token should pay attention to. The transformer is the machine that runs that decision many times over and turns the result into an answer.
How does a transformer work?
Each transformer layer runs self-attention first, so every token gathers information from every other token in the window, then passes the result through a feedforward network that transforms it further. Positional encoding gets added before any of this, since self-attention on its own has no sense of word order. Stacking dozens of these layers lets the model build up increasingly abstract representations, from individual words in early layers to relationships and meaning in later ones, and the final layer turns that representation into a prediction of the next token.
This is also why transformers get expensive at length: self-attention compares every token against every other token, so cost grows quadratically with the size of the context window. Qontext changes what reaches the window in the first place, drawing the relevant information from the context repository so the model spends its attention on what actually matters.