12 Beyond recurrence: Google Brain and the transformer architecture

In this chapter, the transformer is presented as a turning point in how neural networks handle sequence data. Earlier recurrent models processed information step by step, which made them intuitive but also forced distant relationships to pass through repeated state updates and prevented full parallelization during training. The transformer kept the same basic learning framework of prediction, loss, backpropagation, and optimization, but replaced sequential state propagation with self-attention, allowing context to be built from direct relationships among tokens rather than compressed memory passed along a chain.

The chapter then explains how self-attention works through queries, keys, values, dot products, masking, and softmax. Using simplified examples, it shows how a token such as it can weigh surrounding words, produce a contextual representation, and support next-token prediction. The same mechanism is then extended to more complex behavior: semantic association can connect a prompt with learned facts, and autoregressive generation can chain predictions together into longer responses. Causal masking ensures the model only uses information available at each position, while the output layer turns the final contextual state into a probability distribution over possible next tokens.

Finally, the chapter places the transformer in a broader intellectual and practical setting. It argues that the architecture is not a new theory of learning, but a reorganization of established ideas from neural networks, probability, likelihood, and information theory. Its influence comes from the combination of direct relational modeling, multi-head attention, modular blocks, and parallel-friendly computation, which made scaling more effective and opened the door to large language models and other generative systems. At the same time, the chapter stresses limits: high computational cost, finite context windows, imperfect reliability, and the possibility of fluent but wrong outputs. The result is a balanced view of the transformer as a foundational architecture whose core ideas are likely to persist even as future systems evolve.

The transformer architecture introduced in Attention Is All You Need. Stacked encoder and decoder layers combine multi-head attention with feed-forward transformations, residual connections, and normalization. Positional encodings preserve sequence order without recurrence. The decoder uses masked self-attention to prevent access to future tokens and cross-attention to incorporate encoder outputs. Within each multi-head attention block, individual heads use dot products to calculate compatibility scores and softmax to convert those scores into attention weights, as demonstrated in the worked examples that follow. Reproduced from Vaswani et al (2017), Attention Is All You Need, Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017).

• Recurrent neural networks modeled sequence by carrying context through an evolving hidden state. This approach worked well for many tasks but made distant relationships harder to preserve and required token positions to be processed sequentially.
• In 2017, Vaswani and his Google Brain colleagues reorganized sequence modeling around self-attention. Attention already existed, but eliminating recurrence and making attention the architecture’s central operation represented the decisive break.
• Self-attention uses queries, keys, values, compatibility scores, and learned weights to construct context from relationships among tokens. Multi-head attention allows several relational patterns to contribute simultaneously rather than forcing context into one dominant relationship.
• Direct relational access made long-range dependencies easier to preserve, while parallel token processing substantially accelerated training. Repeated transformer blocks provided a modular architecture that could expand systematically in depth, width, data, and parameter count.
• When combined with large data sets, improved training, and greater computation, the transformer helped make practical a major leap in capability. Models could sustain extended text, translate languages, generate code, analyze documents, maintain dialogue, and integrate multiple forms of data with unprecedented breadth and coherence.
• Transformer-based language models nevertheless generate one token at a time. Attention helps construct relevant context, but factual recall and multistep generation depend on all the model’s learned components working together. Even then, fluent output may be factually or logically incorrect.
• The transformer preserved rather than displaced established foundations. Probability continued to represent possible outcomes, likelihood and cross-entropy continued to guide learning, neural networks continued to develop representations through backpropagation, and generalization remained essential beyond the training data.
• Its historical significance lies in combining these accumulated foundations with a new relational architecture suited to modern computation. By pushing outward earlier constraints on context, training, and scale, the transformer expanded the range and complexity of uncertainty that learning systems could address.
• Scalability, adaptability, and accumulated technological investment position the transformer for continued influence, but computational demands, reliability limitations, and emerging alternatives make its future dominance uncertain. Even if the original architecture changes, its central ideas may carry forward into successor systems.

FAQ