5 Attention is all you need
This chapter explains how the Transformer became a turning point in modern AI by replacing recurrent and convolutional sequence processing with attention-centered computation. Earlier language models and seq2seq systems struggled to preserve information across long sequences because they processed tokens step by step and often compressed an entire input into a single fixed-length “thought vector.” The Transformer kept the useful encoder-decoder framing for translation but removed recurrence, allowing every token to directly attend to every other token in parallel. This made it faster to train, better at long-range dependencies, and easier to scale, setting the stage for later systems such as GPT-style large language models.
The chapter breaks down the Transformer’s core mechanisms: tokens become learned embeddings; self-attention uses queries, keys, and values to decide which tokens matter to each other; multi-head attention lets the model learn several kinds of relationships at once; feed-forward layers transform each position; and residual connections plus normalization stabilize deep stacks. Since attention alone has no built-in notion of order, positional encodings are added so the model can distinguish phrases like “fat cat” from “cat fat.” The decoder uses masked self-attention for next-token prediction and cross-attention to consult encoder outputs during translation. Empirically, the Transformer achieved state-of-the-art machine translation results while training more efficiently than recurrent alternatives, showing that recurrence was not essential for strong sequence modeling.
The chapter also situates the Transformer within a broader lineage of attention research. Bahdanau attention solved the seq2seq bottleneck by letting decoders dynamically align with source tokens instead of relying on one compressed vector. Pointer Networks showed that attention could act as an output mechanism by pointing directly to input elements, useful for combinatorial problems such as convex hulls and traveling-salesman-style tasks. Neural Turing Machines explored differentiable memory access through attention-like read and write heads, while “Order Matters” and Sutskever’s input-reversal trick showed how sensitive early sequence models were to ordering and optimization path lengths. The Universal Transformer later tried to reintroduce a form of recurrence, but the simpler, scalable original Transformer proved more influential. The chapter concludes that the Transformer’s success came not from a single idea, but from the powerful combination of attention, parallelism, positional information, modularity, and scale.
A simplified schematic of the Transformer architecture. The model consists of an encoder (left) and a decoder (right), each comprising repeated stacks of sublayers, including multi-head attention mechanisms, feed-forward neural networks, and postprocessing steps such as residual connections and layer normalization. Positional encodings (depicted as circled crosses) are added to the input embeddings to explicitly represent token order. The diagram emphasizes the cross-attention operation (dashed box labeled "Cross Attention"), in which the decoder’s queries (Q) interact with the encoder’s keys (K) and values (V). During cross-attention, each decoding step computes attention scores to determine the relevance of encoded input representations, producing a context vector that guides token generation. This selective alignment enables the decoder to integrate contextual information from the encoder, improving the accuracy and coherence of the output sequence.
The Universal Transformer. Used with explicit permission granted by the lead author, Mostafa Dehghani.
The encoder updates a hidden state as it reads each English word. The final hidden state becomes the fixed-length context or “thought vector.” The decoder uses this vector, plus the Spanish words it has already generated, to predict a probability distribution over the next Spanish word. During training, the encoder and decoder learn these representations from many paired source-target sentences by minimizing the prediction error on target translations.
Word-embedding vector operations: Paris - France + Italy ≈ Rome
BLEU scores versus sentence length: the encoder-decoder scores drop steeply beyond about 30 words, while RNNsearch remains stable up to 60 words. Used with explicit permission granted by the lead author (Dzmitry Bahdanau).
illustrates sample alignments generated by the RNNsearch-50 model. The plot’s x- and y-axis correspond to words in the source (English) and target (French) sentences, respectively. Pixels indicate the weight assigned by the attention mechanism to each source annotation when predicting each target word, visualized in grayscale (0 = black, 1 = white). Each row shows the weights associated with the annotations, indicating which positions in the source sentence were most important when generating each target word. The alignments are largely monotonic, with strong diagonal weights in each matrix. However, several non-monotonic alignments also appear. Adjectives and nouns are often ordered differently between French and English. For example, the phrase “European Economic Area” is translated as “zone économique européenne.” The model aligns “zone” with “Area,” initially skipping over “European” and “Economic,” then revisits preceding words to complete the full phrase. Used with explicit permission granted by the lead author (Dzmitry Bahdanau).
A sequence-to-sequence model (left). The input sequence (A) creates a vector used to generate the output sequence (B). The output dimensionality is fixed by the problem’s dimensionality and remains the same during training and inference.[51] Ptr-Net (right) is also an encoder-decoder. The encoder and decoder hidden states are denoted as (ℎ1,…,ℎ𝑛) and (𝑠1,…, 𝑠𝑛), respectively. Ptr-Net encodes the input sequence into a code (A) that is fed to the generating network (B). At each step, the decoder produces a vector that modulates a content-based attention mechanism over the inputs. The attention mechanism outputs a softmax distribution with a dictionary size equal to the input length. Used with explicit permission granted by the lead author (Oriol Vinyals).
Neural Turing Machine architecture. A controller network takes inputs, produces outputs, and manages an external memory. Read and write heads move over this memory to store and retrieve information.
Read-Process-Write reads an input sequence into memory, processes it with an LSTM-based recurrent controller, and writes an output sequence. The Read module encodes the inputs, the Process module iteratively updates a hidden state while attending to memory, and the Write module generates outputs one token at a time, feeding its own predictions back as inputs. The arrows between the decoder’s outputs highlight the autoregressive loop, where each generated token influences the next. Used with explicit permission granted by the lead author (Oriol Vinyals).
The reversal trick shortens the path from relevant source tokens to the decoder’s first predictions. In a vanilla RNN encoder–decoder without attention (a), the decoder starts from the encoder’s final state. Reversing the source (b) makes tokens that matter for early outputs appear later in the encoder’s processing, so their influence reaches the decoder more directly.
Impact
That is, Transformers are not failing at their purpose; they succeed so well that their outputs can fool us into believing they understand. In this sense, the critique collapses into a paradox: language models are criticized not for failing, but for being too good at the very task for which they were designed. After all, no one said this when Andrej Karpathy wrote about the unreasonable effectiveness of RNNs, celebrating that same probabilistic mimicry. The difference was that RNNs never fooled anyone. Transformers did. The cultural resonance of “stochastic parrots” underscores just how deeply Transformers unsettled expectations.
The shift was profound. Neural machine translation and neural sequence modeling had long been defined by the assumption that recurrence was essential. Researchers even joked during this period about the sudden obsolescence of their hard-won expertise in recurrent neural networks, which attention rendered less relevant.
These constraints set the stage for the Transformer, which discarded both with a bold, provocative claim captured succinctly in its paper title. Rather than relying on recurrence, where information propagates sequentially through hidden states, or convolutions, which compose context through local interactions, the Transformer depends exclusively on self-attention within sequences.
The idea that attention alone, without recurrence, without convolution, without any explicitly sequential inductive bias, could perform complex sequence transduction was more than a technical proposal; it was philosophical. It implied that the structure of language, long assumed to require step-by-step processing or local aggregation, could be captured from the start by nonlocal operations. Not only could it be done, but it could also be done better, faster, and cheaper. The Transformer showed that recurrence was a convenience rather than a necessity and revealed scaling as the primary driver of progress in machine learning. In the end, recurrence and convolution did not require funerals. They put up a fight but ultimately became unnecessary. Attention may not literally be “all you need,” but it was certainly enough to dismantle entrenched assumptions about essential structures in neural sequence modeling.
Among the papers on Sutskever’s List, “Attention Is All You Need” is the most influential. Its core idea became an iconic catchphrase, recognized widely beyond academic circles, and a 2025 Nature analysis ranks it seventh among the most-cited papers of the 21st century. Only one paper on Sutskever’s List outranks it.[83] While residual connections quietly enabled deeper networks, attention redefined our relationship with technology.[84]
Sutskever's List ebook for free
