5 Reward Modeling

Reward modeling is the mechanism that lets reinforcement learning from human feedback learn from human judgments rather than a fixed environment reward. In this setup, a language model is trained to turn prompt-completion pairs into a scalar score or preference probability, compressing complex human notions of quality into a signal that can guide later optimization. The chapter frames this as a proxy objective for alignment: instead of directly specifying what good behavior is, humans provide comparisons or labels, and the model learns to imitate those preferences.

The standard reward model is based on the Bradley-Terry preference framework, which assumes that preferences arise from latent scores and can be modeled through pairwise comparisons. A reward model typically attaches a small linear head to a language model and is trained on chosen versus rejected completions using a contrastive objective that pushes the preferred answer higher than the rejected one. The chapter also notes practical implementation details, such as using the final hidden state or end-of-sequence token to produce the scalar reward, and emphasizes that the core training loss is simple even if the surrounding data pipeline is not.

Beyond the default pairwise preference model, the chapter surveys several reward-model variants used for different tasks. Outcome reward models predict correctness, often at the token level, while process reward models score intermediate reasoning steps instead of only the final answer; both use classification-style losses but differ in supervision granularity. The chapter also compares these with value functions, which estimate expected future return rather than immediate correctness, and with generative reward modeling, where another language model acts as the judge. Overall, the main takeaway is that reward modeling is a family of related but distinct approaches, each designed to capture quality, correctness, or reasoning quality at different levels of detail.

The reward model in RLHF plays the role of the environment component that returns rewards in standard RL. The key difference is that in RLHF, we get to control and learn this reward function from human preferences, rather than having it fixed by the environment.

Training a preference reward model requires pairs of chosen and rejected completions. The model computes a scalar score for each completion from a sequence-level representation, often the end-of-sequence (EOS) token’s hidden state, and the contrastive loss depends only on the score difference between the two.

At inference time, an outcome reward model outputs per-token correctness probabilities. Prompt tokens are masked (e.g., label=-100), while completion tokens each receive a probability indicating whether the model believes the response leads to a correct answer.

Training an outcome reward model uses offline labels from a verifier or dataset (e.g., all 1s for correct completions). Each completion token is trained with binary cross-entropy against the outcome label, and per-token probabilities are aggregated into a final score for verification, filtering, or reranking.

Process reward models provide supervision only at step boundaries (e.g., newline tokens). Each step receives a 3-class label: correct (+1), neutral (0), or incorrect (-1). All other tokens are masked during training.

• Reward models are the component of RLHF that learns to represent human preferences – they compress complex, hard-to-specify human judgments into a scalar signal that downstream optimization can act on.
• The standard reward model for RLHF is derived from the Bradley-Terry model of preferences, trained with a contrastive loss to predict the probability that one completion is preferred over another.
• Beyond preference reward models, two popular variants for reasoning tasks include outcome reward models (ORMs) that predict per-token correctness in a response and process reward models (PRMs) that score individual reasoning steps.
• Reward models are typically implemented by appending a linear head to a pretrained language model (e.g., via AutoModelForSequenceClassification), outputting a single scalar score per sequence. Value functions, covered in the RL chapter, share a similar architecture but differ in what they predict and how they are trained.
• Generative reward models use LLM-as-a-judge prompting to approximate human preferences, enabling cheaper data collection and evaluation. Many other research variants exist – including margin losses, K-wise ranking models, and aspect-conditioned reward models – though none have converged into a single best practice.

FAQ