This appendix serves as a compact glossary for the notation and operations used throughout RLHF, while also giving a short refresher on language modeling. It explains that modern language models are typically trained autoregressively, meaning each token is predicted from the tokens that came before, and that training usually minimizes a negative log-likelihood or cross-entropy objective. It also notes that decoder-only Transformers with self-attention are the dominant architecture, and that the final language modeling head maps internal representations into vocabulary space.
From there, the text defines core machine learning and NLP terms needed to follow RLHF discussions. It introduces KL divergence as a measure of how one probability distribution differs from another, and it clarifies common prompt-and-completion terminology, including chosen and rejected completions, preference relations, and policies over completions. These definitions establish the basic objects used when comparing model outputs and modeling preferences between alternatives.
The appendix then lays out reinforcement learning concepts that connect directly to RLHF, such as rewards, actions, states, trajectories, discounting, value functions, Q-functions, advantage functions, and policy-conditioned expectations. It also defines finite-horizon return, on-policy data, and the reference model used to regularize RLHF optimization. Finally, it extends the glossary with concepts like synthetic data, distillation, knowledge distillation, in-context learning, and chain-of-thought, highlighting how modern language model training and alignment often rely on related techniques beyond standard RL.
FAQ
What does an autoregressive language model predict?A language model predicts the joint probability of a token sequence by generating each next token based on the previous tokens in the sequence.How is the probability of a token sequence factorized in a language model?The sequence probability is written as a product of conditional probabilities: \(P_{\theta}(x) = \prod_{t=1}^{T} P_{\theta}(x_t \mid x_{1}, \ldots, x_{t-1})\).What is the negative log-likelihood loss used for in language modeling?The negative log-likelihood loss is used to maximize the likelihood of training data by minimizing the negative of the log probability assigned to the correct next tokens.What is the difference between negative log-likelihood and cross-entropy loss?In practice, cross-entropy loss is commonly used to measure how well the model’s predicted next-token distribution matches the true token, and it is equivalent to next-token prediction training in language modeling.What is KL divergence?KL divergence measures the difference between two probability distributions. For discrete distributions, it is defined as \(\mathcal{D}_{\text{KL}}(P || Q) = \sum_{x \in \mathcal{X}} P(x) \log \left(\frac{P(x)}{Q(x)}\right)\).What is the difference between a prompt and a completion?A prompt is the input text given to a language model, while a completion is the output text generated in response to that prompt.What do chosen completion and rejected completion mean in RLHF?The chosen completion is the preferred output in a pairwise comparison, while the rejected completion is the less preferred alternative.What is a policy in RL and RLHF?A policy is a probability distribution over possible actions or completions, parameterized by model parameters, such as \(\pi_{\theta}(y\mid x)\) in RLHF.What is the advantage function?The advantage function measures how much better an action is compared to the average expected value of the state, and it is defined as \(A(s,a) = Q(s,a) - V(s)\).What are synthetic data and knowledge distillation?Synthetic data is training data produced by another AI system. Knowledge distillation is a training method where a student model learns from the output distribution of a stronger teacher model, rather than only from selected target outputs.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Reinforcement Learning from Human Feedback ebook for free
