Self-attention
Self-attention is the mechanism that lets a model compare every token in a sequence against every other token to weigh what matters.
Category
Industry concept
See all terms
What is self-attention?
Self-attention is a calculation that lets a model score how much every token in a sequence should influence every other token, then blends them into a new representation weighted by those scores. Older sequence models processed text one token at a time so a word early in a sentence had to survive many steps before it could affect a word much later. Self-attention removes that bottleneck: every token gets a direct line to every other token in the same pass, regardless of distance.
How does self-attention differ from a Transformer?
Self-attention is the mechanism, while a Transformer is the architecture built around it. A Transformer stacks self-attention layers with feedforward layers, normalization, and positional encoding into a full model that can be trained end to end. Self-attention alone just produces the weighted comparison between tokens; it takes the rest of the Transformer's structure to turn that into a working language model.
How does self-attention work?
Each token gets turned into three vectors: a query, a key, and a value. The query from one token is compared against the key from every other token to produce a score, those scores are turned into weights, and the weights are used to blend the values into a new vector for that token. Every token runs this comparison against every other token in the same pass, which is why the cost grows quadratically with sequence length: doubling the tokens roughly quadruples the comparisons. This is also why context windows have a hard ceiling: the self-attention step inside a Transformer is what makes a longer window expensive to run. Qontext's context agents narrow the context repository down to the relevant information before it ever reaches the window, so self-attention only has to compare the tokens that matter.