Reinforcement learning from human feedback arose as a way to train models when it is difficult to hand-design a reward function, and its central challenge is that it tries to capture human preferences rather than a clean objective with a single correct answer. The chapter uses simple comparisons, like judging poems, to show that preference judgments are often ambiguous, subjective, and dependent on context, unlike factual questions with a definite solution. Because of this, RLHF is fundamentally about building an indirect signal that can guide models toward behavior people like, even when there is no obvious ground truth.
The text traces RLHF to several intellectual traditions at once: philosophy, psychology, economics, decision theory, reinforcement learning, and deep learning. It explains how ideas such as utility, reward, cost, and expected value developed over time, and how modern reinforcement learning inherited assumptions from control theory and Markov decision processes that work well in closed, well-specified environments. Those assumptions fit tasks with stable rewards and clear optima, but they sit uneasily beside human preferences, which are more variable, contextual, and difficult to compress into a single scalar signal.
The chapter’s main argument is that this mismatch creates many practical and theoretical complications. Preferences can shift over time, differ across people, depend on presentation, and resist simple aggregation, while social choice results and utility theory reveal limits on what any one model can represent. In RLHF, reward models therefore become imperfect proxies that must summarize messy, changing, multidimensional judgments, and the process is shaped by assumptions that may not hold in real settings. The result is not that RLHF is useless, but that it is inherently incomplete and unlikely to be fully solved, since it sits at the intersection of human values and optimization systems with different underlying logics.
The timeline of the integration of various subfields into the modern version of RLHF. The direct links are continuous developments of specific technologies, and the arrows indicate motivations and conceptual links.
Summary
RLHF sits at the intersection of philosophy, economics, psychology, reinforcement learning, and deep learning – each bringing its own assumptions about what preferences are and how they can be optimized.
Reinforcement learning was designed for domains with stable, deterministic reward functions, but human preferences are noisy, context-dependent, temporally shifting, and not always transitive – a fundamental mismatch that shapes the limitations of RLHF.
The Von Neumann-Morgenstern utility theorem provides theoretical license for modeling preferences as scalar functions, but its assumptions (transitivity, comparability, stability) are routinely violated in practice. Impossibility theorems in social choice theory further show that no single aggregation method over preferences can satisfy all fairness criteria simultaneously.
These challenges explain why RLHF will never be fully “solved,” but they do not prevent it from being useful. In practice, RLHF operates on more tractable problems of style and performance rather than attempting to resolve the full complexity of human values.
The practical mechanics of collecting and structuring preference data in light of RLHF’s complex motivations are covered in Chapter 11.
FAQ
What is the central idea of “preferences” in RLHF?Human preferences are the target signal RLHF tries to model. Instead of hand-designing a reward function, RLHF uses human judgments to approximate what people consider better, more helpful, or more aligned with their values.Why is RLHF considered an interdisciplinary problem?RLHF draws on philosophy, psychology, economics, decision theory, reinforcement learning, and deep learning. Each field contributes different assumptions about what preferences are, how they can be measured, and how they should be optimized.How does RLHF differ from ordinary reinforcement learning with a fixed reward function?Traditional RL usually assumes a stable, well-defined reward or cost function in a structured environment. RLHF instead tries to learn from human feedback, where the target is less explicit, more subjective, and often context-dependent.Why does the chapter say RLHF will never be a fully solved problem?Because human preferences are not perfectly stable, universal, or easy to encode in a single objective function. They can vary across people, contexts, and time, so any practical system must rely on imperfect assumptions and approximations.What assumptions from reinforcement learning become problematic in RLHF?RL assumes rewards are stable, deterministic, and can be expressed as a function of state-action pairs. In RLHF, preferences may drift, depend on presentation or context, and be difficult to reduce to one scalar reward.Why is the idea of “reward-to-go” important in this chapter?Reward-to-go is a core RL concept that summarizes future expected reward from a given point. The chapter uses it to show how RL was built for clear optimization tasks, which contrasts with the messier nature of human preferences.What role does the Von Neumann-Morgenstern utility theorem play in the discussion?The theorem provides a formal foundation for modeling preferences as expected utility under uncertainty. The chapter explains that while this is theoretically useful, its assumptions often do not hold cleanly in RLHF practice.Why is the term “value” confusing in the context of RLHF?In reinforcement learning, “value” is a numerical estimate used for optimization. In discussions of human values, it refers to ethical or moral principles, which are broader and much harder to define mathematically.What kinds of real-world factors can distort preference measurements?Choices can be affected by visual presentation, task context, sequential labeling effects, user differences, and implicit signals like page views or tab closing. These factors mean preference data may reflect more than just stable underlying likes or dislikes.What is the main takeaway of the chapter for RLHF dataset and model design?The key lesson is that preference modeling is an approximation, not a perfect recovery of human values. Designers must account for ambiguity, shifting preferences, and social or contextual effects when building RLHF systems.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Reinforcement Learning from Human Feedback ebook for free
