Reinforcement learning from human feedback is a very recent family of methods, and the chapter uses its history to show how quickly it moved from niche academic ideas into a core technique for training modern language models. Rather than presenting a complete survey, it focuses on the papers and projects that most directly shaped the pipeline used today, especially the developments that eventually fed into ChatGPT. The emphasis is on how the field’s documentation lives largely in research papers, while the underlying ideas have remained stable even as details continue to evolve.
The earliest phase traces back to reinforcement learning from preferences, where humans guided agents by comparing actions or trajectory segments instead of providing direct environment rewards. Early examples such as TAMER and COACH showed that evaluative human input could be turned into a learned reward signal or advantage adjustment, and Christiano et al. later made this idea especially influential by training reward models from trajectory comparisons in Atari games. This period established the basic loop that still defines RLHF: collect human judgments, fit a reward model, and optimize a policy against that learned signal, while later work began to frame reward models more broadly as tools for alignment rather than only for solving control tasks.
From 2019 to 2022, the approach was adapted to language models and became the foundation for systems like InstructGPT, WebGPT, GopherCite, Sparrow, and early Claude work. The 2019 language-model paper helped define canonical practices such as reward modeling from preferences and KL penalties to keep model updates restrained, and subsequent research explored key issues like reward over-optimization, using language models for alignment, and red teaming for safety evaluation. In the ChatGPT era, RLHF became widely associated with leading chat models and has since expanded into broader preference fine-tuning, including process reward, direct alignment methods like DPO, and other feedback-driven approaches for reasoning, coding, and math.
Timeline of key developments in RLHF discussed in this chapter, from early work on RL from preferences through the adoption of RLHF in large language models.
The core RLHF loop from Christiano et al. (2017): the reward predictor is trained asynchronously from comparisons of trajectory segments, and the agent maximizes predicted reward.
Summary
RLHF techniques used today are deeply related to early RL from preferences literature, including Knox & Stone’s TAMER (2008) and Christiano et al. 2017, which applied RLHF to Atari games using human preferences between trajectories.
Ziegler et al.’s 2019 paper “Fine-Tuning Language Models from Human Preferences” formalized many canonical terms still used today when applying RLHF to LLMs: reward models, KL divergence for regularization, and the basic feedback pipeline.
Before ChatGPT, RLHF was applied to summarization (Stiennon et al.), instruction following with InstructGPT (Ouyang et al.), browser-assisted question answering with WebGPT (Nakano et al.), and dialogue with Sparrow (Glaese et al.).
ChatGPT’s announcement explicitly credited RLHF for its success, and since then RLHF has been used extensively in leading models including Claude, Llama 2/3, Nemotron, and Tülu 3.
FAQ
What is the main purpose of Chapter 2, “A Tiny History of RLHF”?The chapter gives a brief historical overview of RLHF, showing how the field developed over time and highlighting the key papers and projects that shaped modern RLHF, especially the path that led to ChatGPT. It is not meant to be a complete survey, but a focused retelling of the most influential developments.Why does the chapter say RLHF is still a very new field?RLHF and related methods were formalized only recently, and much of the important documentation is still found in academic papers rather than textbooks. The chapter emphasizes that some details may change over time, but the core practices are already well established.What are the three eras of RLHF described in the chapter?The chapter divides RLHF history into three periods: origins to 2018, when research focused on RL from preferences; 2019 to 2022, when RLHF was adapted heavily for language models; and 2023 to the present, the ChatGPT era, when RLHF became central to major chat models.What early work is considered similar to modern RLHF?One of the earliest similar approaches was TAMER, which used human evaluative feedback to learn a reward model and then a policy. Another related method was COACH, which used human positive and negative feedback to tune the advantage function.Why is Christiano et al. (2017) so important in RLHF history?Christiano et al. (2017) is treated as a primary reference because it applied RLHF to preferences between trajectory segments in Atari games. It helped establish the core RLHF loop: train a reward predictor from human comparisons, then optimize the agent against that learned reward.What did the 2019 paper “Fine-Tuning Language Models from Human Preferences” contribute?This paper brought RLHF-style methods directly to language models and introduced many canonical ideas used today, including reward models trained on human preferences, KL divergence to keep models from changing too much, and the overall feedback pipeline. It is one of the most influential early works connecting RLHF to LLMs.How did RLHF expand between 2019 and 2022?RLHF was applied to many language-model tasks, including summarization, recursive book summarization, instruction following, browser-assisted question answering, citation-supported answers, and open-ended dialogue. These applications helped make RLHF a standard tool for aligning LLM behavior.What key research directions emerged from RLHF during this period?The chapter highlights three major directions: reward model over-optimization, language models as a broader alignment target, and red teaming as a way to evaluate model safety. These areas influenced later work on robustness and alignment.How did ChatGPT change the public understanding of RLHF?The ChatGPT announcement explicitly stated that the model was trained using RLHF, using methods similar to InstructGPT with some differences in data collection. This made RLHF widely known outside research circles and strongly associated it with modern chatbots.What is RLHF evolving into today?Today, RLHF is expanding into a broader preference fine-tuning landscape. This includes process reward for intermediate reasoning, direct alignment methods like DPO, learning from execution feedback in code or math, and other reasoning-focused approaches inspired by systems like OpenAI’s o1.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Reinforcement Learning from Human Feedback ebook for free
