The transformer is a neural network design introduced in the 2017 paper "Attention Is All You Need". Its key mechanism, self-attention, lets each token in a sequence look at every other token and decide which matter most for its meaning.
Why it matters
Transformers process sequences in parallel, which trains efficiently on GPUs and scales to very large models. They underpin LLMs, many image and audio models, and translation systems. The design also made it practical to pre-train one large model and adapt it to many tasks.
Example
In the sentence "The server crashed because it ran out of memory", attention helps the model link "it" to "server" rather than "memory". Real models stack many attention layers, each with multiple heads that capture different relationships.
Common mistakes
- Confusing the electrical device or the toy line with the architecture; the name is unrelated.
- Assuming the cost of attention is constant. It grows quickly with sequence length, which is one reason the context window is limited.
- Believing every modern AI model is a transformer. Other architectures remain in use.
Related
Get notified when this term updates
Plus occasional practical tips on uptime and monitoring, nothing else. Unsubscribe anytime.
This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.