11 Preference Data

Preference data is the central signal that powers RLHF, because human values and model behavior are too complex to compress into a single hand-designed reward function. Instead of trying to define perfect rewards, RLHF relies on labeled comparisons that reveal which outputs people consider better, safer, clearer, or more helpful. The key intuition is that it is usually much easier for humans to judge between two candidate answers than to produce an ideal answer from scratch, making preference labels a practical proxy for aligning models.

Collecting this data is expensive and operationally difficult, often requiring large budgets, careful instructions, iterative refinement, and either specialized vendors or in-house annotators. A major assumption is that the most useful data is on-policy, meaning it comes from generations produced by the current family of models rather than unrelated model outputs. The chapter also describes how preference can be gathered through different interfaces, from pairwise comparisons and rating scales to thumbs-up/thumbs-down feedback, multi-turn conversations, and structured or synthetic examples where correctness can be checked automatically.

At the same time, the chapter emphasizes that preference data is noisy and vulnerable to subtle biases such as prefix bias, sycophancy, verbosity, and formatting effects. These issues can be hard to detect but can strongly shape the resulting model if left unaddressed. The chapter closes by highlighting open questions about who collects the data, what kind of feedback is best, how demographic and workplace context influence preferences, and whether the preferences written into the data are truly the ones reflected in the final models.

An example of one of the earliest preference data collection interface, from Anthropic’s research. Bai et al. 2022. The actual conversation is a toy conversation around what is a good example conversation for data collection. License CC-BY.

Example preference data collection interface from when I was served two completions from different ChatGPT beta models. The actual completions are very close in content, showing how collecting preference data can be noisy and difficult to get exactly right.

Example preference data collection interface from an early version of the popular Arena benchmark.

Example preference data collection interface with thumbs up or thumbs down icons from the Allen Institute of AI’s research demos.

Example user interface of text-to-image models.

Overview of the multi-batch cycle for obtaining human preference data from a vendor. The ramp up period allows a narrowing of goals and methodology in order to create the best possible data. It is expected that a larger proportion of the data from the earlier batches will have to be thrown out due to quality issues. This is one timeline example for a smaller data contract (~$500K) and much larger data contracts can vary substantially.

• Preference data is the primary engine of RLHF improving models – because we cannot write precise reward functions for human values, pairwise comparisons between model completions serve as the proxy signal for training.
• Effective preference data should be on-policy: collected on completions from the model being trained (or a closely related checkpoint), so the training signal covers the token distributions the model actually generates.
• Collecting human preference data is operationally complex and expensive, involving expensive, multi-week vendor contracts, iterative calibration of annotation guidelines, and careful integration with training pipelines – which is why many teams opt for synthetic preference data instead.
• Preference data is prone to subtle biases – including verboseness bias, sycophancy, prefix bias, and formatting preferences – that can transfer directly to the trained model if not detected and mitigated.
• Core open questions remain around data collection contexts, whether pairwise comparisons are the right format for capturing preferences, population demographics of annotators, and whether the behaviors specified in annotation guidelines actually manifest in the final models.
• Chapter 12 covers how synthetic data and AI feedback complement and extend human preference data across the pipeline.

FAQ