12 Synthetic Data

Reinforcement learning from human feedback began as a human-centered process because people were the only reliable source of high-quality answers and preference labels. As language models improved, however, synthetic data became practical and increasingly central: it is cheaper, easier to iterate on, and now powers much of modern post-training. The chapter explains that synthetic data is not a marginal substitute but a major ingredient in current pipelines, especially for generating prompts, completions, feedback, and filtered training examples.

A key concern with synthetic data is model collapse, where repeated self-training can reduce diversity and amplify errors. The text argues that this risk is mainly associated with narrow, repetitive self-generated data, and that it is much less severe when synthetic data is mixed with human data, produced by multiple teachers, deduplicated, and filtered carefully. It also emphasizes that today’s strongest models often outperform humans at producing certain kinds of training data, making synthetic data essential for instruction tuning, preference data generation, and large-scale evaluation.

The chapter then surveys the main ways synthetic data is used. Distillation transfers knowledge from stronger models to weaker ones, either broadly across post-training or for targeted skills like reasoning and coding. AI feedback and constitutional methods use language models to critique, rank, or score outputs, lowering cost while preserving useful supervision, though human data still matters for hard cases and trustworthy ground truth. Finally, rubric-based training extends these ideas by giving models prompt-specific criteria for judging responses, showing how synthetic supervision is expanding into more complex forms of alignment and reinforcement learning.

Traditional knowledge distillation trains a smaller student model to match the soft probability distribution of a larger teacher model using KL divergence loss. Both models process the same input simultaneously, and temperature scaling (\(\tau > 1\)) softens the distributions to reveal more information about class relationships.

Synthetic data generation in LLM post-training: prompts are passed through a strong model to generate completions, which are paired to create a training dataset. This dataset is then used to fine-tune smaller models via standard supervised learning. More complex pipelines may involve multiple models editing completions, generating preference pairs, or filtering for quality.

• Synthetic data has become essential to modern post-training – language models are used to generate training prompts, write completions, provide preference labels, and filter data at every stage of the pipeline.
• Distillation, using outputs from a stronger model to train a smaller one, is the most common form of synthetic data usage and has largely replaced human-written completions for instruction tuning, though human data remains important at capability frontiers and for establishing ground truth answers.
• AI feedback can approximate human preference labels at a fraction of the cost within the RLHF pipeline, denoted RLAIF. A useful rule of thumb: human data is high-noise and low-bias, while synthetic preference data is low-noise and high-bias – making AI feedback easier to start with but prone to systematic second-order effects.
• Constitutional AI (CAI) was the earliest large-scale use of synthetic data for RLHF, using a set of written principles to guide both instruction data revision and synthetic preference data generation.
• Rubric-based rewards extend RL training to domains without clearly verifiable answers by using LLM judges to score completions against per-prompt evaluation criteria, enabling RL-style training on tasks like scientific reasoning, creative writing, or model personality.

FAQ