4 Large language models

Large language models matter not because they introduced a new theory of intelligence, but because they are the most ambitious effort yet to produce intelligent-seeming behavior through computation. Their success depends on the Transformer architecture, which replaced sequential processing with attention across whole sequences, making it possible to capture long-range relationships, add order through positional encodings, and refine representations through self-attention, multi-head attention, residual connections, and normalization. These design choices made deep, stable training feasible and helped turn language modeling into a flexible system for generating coherent text step by step.

What makes these models “large” is not just that they have many parameters, but that they are trained on vast and diverse data with enough compute to exploit that capacity. Scale increases the range of patterns a model can represent, and broad exposure helps it learn regularities across domains, while scaling laws show that performance improves when model size, data, and computation grow together. But scale also changes the architecture of the field: dense growth becomes expensive and inefficient, so modern systems increasingly rely on selective specialization such as mixtures of experts, and on multimodal learning that aligns text with images, audio, and video in shared representational spaces. Even then, these systems remain statistical, learning correlations rather than causation or genuine understanding.

As a result, the chapter emphasizes that the power of large language models comes with heavy practical and social costs. Pretraining, fine-tuning, distillation, quantization, and expert-based routing all aim to make these systems usable without abandoning the scale that gives them capability. Yet training and inference demand immense compute, energy, and infrastructure, which concentrates control in a small number of organizations and turns scale into a gatekeeper for who can build, study, or deploy frontier models. The central lesson is that largeness enables capability, but it also creates dependency, cost, and inequality, so the question is not only what these models can do, but who gets to shape them.

The original Transformer architecture, introduced in 2017. At first glance, the diagram may appear complex, but it is built from a small set of repeating ideas. The rest of this section breaks it down, explaining how positional information, attention, feedforward layers, normalization, and architectural constraints combine to produce the architecture shown here.

Adding positional information to token embeddings. Each token embedding captures meaning but not order, so a positional encoding is added to produce a representation that combines both. This ensures that the same token can have different representations depending on its position in the sequence, enabling the model to account for order.

A single token representation is transformed by three learned projections into a query, key, and value, which serve different roles in attention. The query represents what the token looks for in context, the key what makes it potentially relevant to other tokens, and the value the information it can contribute.

The query for “rises” is compared with the keys of other tokens to produce relevance scores. A stronger match with “sun” gives it a higher score than “the,” indicating that it is more useful for interpreting “rises.”

The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

How multiple heads combine their views. Each attention head applies the full attention mechanism described previously to the same input sequence, producing its own set of hidden states (one contextualized vector per token). These outputs are then placed side by side (concatenated, meaning that for each token, the vectors from all heads are joined into a single longer vector) and transformed through the output projection Wᵒ. This final step blends the contributions of all heads into a unified representation that is passed to the next stage of the model.

Residual connections allow each layer to build on the existing representation by adding its transformation to the input rather than replacing it. This preserves information across layers and makes deep architectures easier to train.

The role of normalization in stabilizing learning. As signals pass through many layers, their values can gradually drift, making learning unstable. Normalization counters this drift by restoring a consistent scale, allowing each layer to receive inputs that behave predictably during training.

Encoder–decoder structure for next-token prediction. The encoder processes the full input sequence to build contextualized representations. The decoder receives a shifted version of the sequence, so that at each position it can attend only to previous tokens, and predicts the next token accordingly. This setup allows the model to learn from complete sequences while enforcing the causal structure required for generation.

Unmasked and masked self-attention. In these matrices, darker cells indicate stronger attention between tokens, while hatched cells mark positions that are masked. In unmasked attention (left), each token can attend to any other token in the sequence. In masked attention (right), future positions are hidden, allowing each token to see only itself and earlier tokens. This causal structure enables step-by-step text generation, where each new token is produced using only what came before.

Cross-attention in a Transformer decoder. In cross-attention, queries are derived from the decoder’s current state, while keys and values come from the encoder’s representations of the full input sequence. This allows each decoding step to match what it is looking for, such as a location completing “rises in the …,” against all positions in the input. In this example, the representation for “east” provides the strongest match and contributes most to the next token prediction. Because the decoder cannot attend to future tokens in its own sequence, it must retrieve this information from the encoder, preserving the causal structure required for generation.

Transformer model families. Encoder-only models process full sequences for understanding tasks. Encoder–decoder models map an input sequence to an output sequence. Decoder-only models generate text step by step and form the basis of modern large language models.

Pretraining and model adaptation. Models are first trained on broad data, then specialized through full or partial parameter updates.

Low-Rank Adaptation (LoRA). A pretrained linear layer with frozen weights is augmented by a small, trainable correction implemented through low-rank projections. The outputs are combined to modify behavior without altering the original parameters.

Knowledge distillation. A large pretrained teacher guides a smaller student model, which learns by matching the teacher’s output distribution rather than relying only on hard targets.

Quantization of model weights. High-precision floating-point weights (left) are rescaled and rounded into lower-precision representations (right), reducing memory and computation while preserving the model’s structure.

Structure of a Mixture of Experts layer. A gating network selects which experts should process each input and assigns a weight to their contributions. The chosen experts run in parallel, and their outputs are combined into a weighted sum. Experts that are not selected for a given input, such as the grayed-out Expert 3, remain inactive. By activating only a subset of experts at a time, the model increases its capacity without increasing the computation required at runtime.

Learning a shared multimodal representational space through contrastive alignment. Text, images, and audio are mapped into a shared space. Related inputs across modalities are pulled closer together, while unrelated inputs are pushed apart, allowing cross-modal associations to be learned through relative distance.

Early and late fusion strategies in multimodal models. In late fusion, each modality is processed mostly independently, and the representations are combined only near the output. In early fusion, modality-specific encoders produce representations that are merged earlier and then processed jointly by shared layers. The choice of fusion strategy determines how deeply modalities can influence one another during learning, trading simplicity and modularity for richer cross-modal interaction.

Cross-modal attention as selective querying. Representations from different modalities are processed within a shared attention mechanism. In this example, image representations act as queries, attending to textual representations that provide keys and values. This reflects a common interaction in multimodal systems, such as using an image to guide a textual response. The resulting attention output updates the model’s hidden state, showing how one modality can retrieve relevant information from another without collapsing them into a single stream.

• Transformers replaced recurrent and convolutional architectures by making attention the central mechanism for processing language, enabling fully parallel sequence modeling.
• The Transformer is built from a reusable architectural block whose behavior depends on how attention is constrained, allowing the same structure to support encoding, decoding, and generation.
• Self-attention allows each token to selectively integrate information from any other position in the sequence, removing distance as a limiting factor for context.
• Multi-head attention provides multiple concurrent views of the same input, enabling the model to capture different relational patterns within language.
• Because self-attention has no inherent notion of order, positional information must be introduced explicitly to anchor tokens within a sequence.
• Residual connections and normalization layers make deep Transformer stacks trainable by preserving information flow and stabilizing learning at scale.
• A language model becomes “large” through the interaction of parameter capacity, broad data exposure, and architectures that remain stable and efficient as scale increases.
• Pretraining allows large models to internalize general linguistic structure, while post-training techniques such as LoRA, distillation, and quantization adapt or compress that structure for practical use.
• Beyond a certain point, scaling uniformly becomes inefficient, and Mixture of Experts layers expand capacity by selectively activating specialized components rather than increasing computation everywhere.
• Text encodes vast amounts of human experience indirectly, allowing language models to appear knowledgeable without directly perceiving the world.
• Tasks that rely on visual, auditory, or temporal grounding expose the limits of text-only models and motivate the integration of additional modalities.
• Multimodal models align text, images, audio, and video within a shared representational space, enabling cross-modal reasoning through attention rather than translation into words.
• Despite broader inputs, multimodal models still learn through co-occurrence and correlation, not through direct interaction with or intervention in the world.
• The development and deployment of large language models are shaped by substantial economic and energy costs, with training representing the dominant upfront investment.
• These costs constrain experimentation by favoring high-commitment training efforts and incremental refinement over broad exploratory research.
• As scale increases, control over frontier-model development becomes concentrated among actors with access to sustained compute and infrastructure, creating a structural gatekeeping effect.

FAQ