2 Training large language models: Learning at scale
Large language models are trained at internet scale to predict the next token, and this chapter explains how that simple objective, combined with massive data and compute, yields sophisticated capabilities. It traces the end-to-end pipeline for both text-only and multimodal systems, from foundational pre-training to specialized post-training and even inference-time strategies that boost performance without retraining. Along the way, it highlights architectural and efficiency innovations that let models grow in capability without costs rising proportionally, and it examines the surprising, sometimes unpredictable behaviors that emerge at frontier scales.
Training begins with self-supervised pre-training on vast, noisy open-web corpora, which equips models with broad world knowledge but also reflects language and geographic skews. Autoregressive models specialize in generation, while bidirectional models focus on understanding; multimodal models extend these ideas by encoding text and images separately and aligning their representations (for example via contrastive learning) to support cross-modal tasks. After pre-training, models are adapted and aligned through supervised fine-tuning, reinforcement learning from human feedback, direct preference optimization, and increasingly AI-based feedback, which together shape behavior toward task goals and human (or AI) preferences.
Efficiency at scale is achieved through knowledge distillation and other compression methods, and through architectures such as Mixture-of-Experts with sparse routing that activate only a small subset of parameters per input. New reasoning-focused approaches further improve results at inference by allocating more “thinking” time and encouraging step-by-step solutions. At very large sizes, models exhibit zero- and few-shot generalization and other apparent emergent abilities; while some argue these effects partly reflect evaluation choices, overall performance continues to rise with scale across data, model size, compute, and smarter architectures, amplified by post-training and inference-time techniques.
The high-level training process for LLMs
Comparison of unimodal and multimodal LLMs
Text and image inputs are processed by separate encoders to generate embeddings. These representations can then be aligned in a shared space during training.
During contrastive learning, similar image–text pairs are encoded close together in a shared embedding space, while dissimilar pairs are mapped farther apart.
The general setup for knowledge distillation
A visualization of the Mixture of Experts layer of a neural network
A typical setup for reinforcement learning from human feedback, after a supervised fine-tuning step for dialogue
A comparison of fine-tuning, zero-shot learning, and one-shot learning on a machine translation task.
Summary
- LLMs might be trained on a combination of open source or public datasets, datasets purchased from third-party vendors, datasets that companies collect by scraping the web, or datasets that the companies create by writing examples for the models to learn from.
- Autoregressive models refer to the fact that each prediction depends only on the tokens that come before it in the input sequence. All the models in the GPT family and Google’s PaLM are autoregressive models trained to predict the next token given some input.
- Multimodal models support multiple input or output modalities. They are trained in similar ways to LLMs but must integrate representations of concepts across modalities, such as text and image.
- Knowledge distillation helps small models learn some of the skills of larger models. Sparsity is a strategy to use larger models more efficiently by only using parts of the model simultaneously, such as in sparse Mixture of Experts.
- LLMs might be post-trained through several techniques, which differ based on their objectives and data.
- Zero-shot and few-shot refer to the number of examples that the model is given before being asked to perform a task. They are primary examples of LLMs' emergent abilities.
FAQ
How are large language models (LLMs) trained at a high level?
LLMs are built in two phases. First, pre-training on massive corpora with a self-supervised token prediction objective (typically autoregressive next-token prediction). Second, post-training adapts the base model to specific uses—such as instruction following, dialogue, or coding—using methods like supervised fine-tuning (SFT), RLHF, DPO, or RLAIF. At scale, models often display emergent abilities beyond what was explicitly trained.
What data sources are used, and what biases do they introduce?
Training mixes open web data (e.g., Wikipedia, Reddit, Common Crawl), curated datasets (news, reviews, Q&A like Stack Exchange), purchased/licensed corpora, and human-authored or filtered data. Because much of the web is English and Western-centric, models tend to perform best in English and often reflect North American/Western European perspectives. Licensing constraints can also affect permissible use.
What’s the difference between autoregressive and bidirectional token prediction?
Bidirectional models (e.g., BERT) learn to fill masked tokens using both left and right context, which is great for understanding tasks. Autoregressive models (e.g., GPT, PaLM) predict the next token using only left context, which is essential for coherent text generation. Choice depends on whether you prioritize understanding or generation.
How do multimodal LLMs learn from text and images?
They use separate encoders (e.g., CNNs/ViTs for images and transformers for text) to produce embeddings, then align them—often via contrastive learning on paired image–text data (as in CLIP). Some models use generative approaches (e.g., DALL·E). Increasingly, unified architectures (e.g., GPT‑4V) jointly process modalities, enabling tasks like visual Q&A. Training relies on paired datasets such as MS‑COCO, Conceptual Captions, and LAION‑5B, followed by task-specific fine-tuning.
How does knowledge distillation make models smaller and faster?
A large “teacher” model generates high-quality examples that a smaller “student” model learns to mimic, often matching teacher performance on targeted tasks with far lower cost and latency. Distillation complements other compression methods like quantization (reduced precision) and pruning (removing low-impact parameters). Well-targeted distillation can rival or beat larger models on domains such as math and coding.
What is a Mixture of Experts (MoE), and why is sparsity useful?
MoE models contain many specialist sub-networks (“experts”) and a learned gate/router that selects a small subset per input. Sparse activation lowers inference cost and latency and can reduce overfitting by limiting active parameters. Training requires balancing expert utilization and stable routing. Many frontier models likely use MoE to scale total parameters without proportional compute at inference.
What are “reasoning models” and inference-time scaling (e.g., o1/R1)?
Reasoning models are trained—often with reinforcement learning—to generate chains of thought (CoT) before answering, spending more “thinking” tokens at inference to improve accuracy on hard problems. OpenAI’s o1 hid CoT tokens; DeepSeek’s R1 surfaced them, showing analysis and self-critique. CoT improves results, though research suggests these traces may not perfectly reflect internal reasoning.
What post-training techniques are used, and when?
- Supervised fine-tuning (SFT): Efficiently teaches desired behaviors with input–ideal output pairs; great for instruction following and classification.
- RLHF: Trains models to prefer human-rated responses using a learned reward model; robust and flexible but more complex.
- Direct Preference Optimization (DPO): Trains directly on human preference pairs; simpler and compute-efficient, strong in many cases.
- RLAIF: Uses AI feedback or verifiers instead of humans to scale alignment cheaply; often combined with human validation.
What are emergent properties, and how do zero-/few-shot prompting help?
Emergent properties are qualitatively new abilities that appear in large models but not smaller ones (e.g., better arithmetic or broad knowledge Q&A). LLMs often perform novel tasks zero-shot, and further improve with a few in-prompt examples (few-shot). Chain-of-thought prompts (“Let’s think step by step”) and asking for confidence or detailed instructions can boost reasoning accuracy.
Is “emergence” real or an evaluation artifact?
There’s debate. Some studies argue sharp performance jumps can stem from evaluation choices (metrics, test sizes, model coverage) rather than a fundamental behavioral phase change. Regardless, maximum performance across tasks keeps rising with scale and better training, so cautious interpretation and rigorous benchmarking are advised.
Introduction to Generative AI, Second Edition ebook for free
