8 Alignment through principles: Constitutional AI

Constitutional AI is presented as a response to the limits of reinforcement learning from human feedback: human preference labels are costly, slow to refresh, inconsistent across annotators, and often biased toward harmless refusals that reduce usefulness. The chapter frames alignment as matching how agents rank outcomes, then argues that written principles are a better training signal than pairwise preferences because they make the intended behavior explicit and reusable across many situations, including ones labelers never saw. This is especially important for real products, where new domains and edge cases keep appearing faster than human labelers can keep up.

The core recipe replaces per-comparison human labeling with an AI critic guided by a constitution, a small set of plain-language principles such as being helpful, honest, and harmless. In the first phase, the model generates responses, critiques them against randomly sampled principles, and revises them; the revised answers become supervised fine-tuning data. In the second phase, the model compares candidate responses using constitution-conditioned AI preferences, trains a reward model on those preferences, and then runs the usual PPO loop. The result is a scalable form of AI feedback that preserves the overall RLHF machinery while shifting the labor from labeling individual examples to writing and maintaining a concise, testable constitution.

The chapter then broadens the idea into a practical framework for policy and product alignment. OpenAI’s Model Spec is treated as a more production-ready version of constitution-as-code, with a chain of command that clarifies which instructions override others and with clauses that can be turned into tests. It also introduces model spec midtraining as a way to reduce shallow alignment by teaching the model the reasons behind the spec before fine-tuning on demonstrations. The main takeaway is that Constitutional AI is powerful, but not complete: it works best when paired with red teaming, calibration, and ongoing spec maintenance so the written rules stay aligned with changing products and model behavior.

The Constitutional AI two-phase pipeline. Phase 1 generates critique-and-revise traces from a helpful-only baseline guided by the constitution, then fine-tunes the pretrained model on the <prompt, revised responses> pairs to produce the SL-CAI model. Phase 2 samples paired responses from the SL-CAI model, asks the model to pick the preferred response under the constitution, trains a reward model on those AI preferences, and runs PPO with a KL penalty against the SL-CAI reference. The downstream RL machinery is identical to InstructGPT.

This chart shows harmlessness vs helpfulness Elo scores (higher is better) computed from human labelers’ model comparisons. It shows a Pareto improvement where Constitutional AI is both more helpful and more harmless than standard RLHF.

The OpenAI Model Spec chain of command. Five nested authority levels, from Root at the center out through System, Developer, User, and Guidance, surrounded by an outermost band for content that carries no authority (assistant and tool messages, quoted or untrusted text). A higher-authority level overrides every level outside it. Content in the outer no-authority band is treated as data, never as instructions, unless a higher level explicitly delegates authority to it.

Model spec midtraining inserted between pretraining and alignment fine-tuning. The pretrained model first trains on synthetic documents about the spec (articles, dialogues, examples). The midtrained model then proceeds through the standard alignment fine-tuning steps (SFT on critique-revised demonstrations, optionally RLAIF). The midtrained model generalizes the spec to held-out prompts in ways models without MSM do not.

Model spec midtraining data generation pipeline; it takes a model spec and generates data using a teacher model to write out background documents for MSM.

Where each alignment technique acts in the model lifecycle. Model spec midtraining acts between pretraining and post-training, on documents about the spec. RLHF, Constitutional AI / RLAIF, and deliberative alignment are done during post-training time, all split into the SFT and RL phases. Deliberative alignment enables the model to reason over principles while generating.

• RLHF is bounded by the human labeler workforce, which does not scale. The shift from preferences to principles changes where the signal comes from.
• A human preference label has a low signal to tell the reward model why one response is better than another.
• A principle is a written statement of what is wanted. An AI critic can apply it to any output, including outputs no human labeler ever saw, and generate as many critiques and preferences as the inference budget allows.
• Alignment is the degree of overlap between the way two agents value different outcomes. For example, if agent A completely internalizes the desires of agent B—i.e., the only desire A has is to see B’s desires satisfied—we could say that agent A is maximally aligned with agent B.
• Alignment targets three axes at once—helpful, honest, and harmless (HHH). Human labelers optimizing for harmlessness prefer evasive refusals because evasion is never penalized, so the reward model learns evasion-as-harmlessness and the policy ends up less helpful than intended.
• Constitutional AI is a two-phase recipe that replaces the human labeler with an AI critic guided by a written constitution.
• SL-CAI: This maps to the supervised fine-tuning (SFT) stage of Instruct GPT. It generates revised responses based on randomly drawn principles; the pretrained model is fine-tuned on the revised responses.
• RL-CAI: This maps to the RLHF stage of Instruct GPT. It trains a reward model based on AI preferences conditioned on the constitution instead of human preferences, followed by PPO in RLHF.
• RLAIF (reinforcement learning from AI feedback) is a one-line substitution into the RLHF pipeline. The reward model and PPO loop are unchanged. Only the source of the preference signal changes from human preferences to AI labeler preferences conditioned on the constitution. Preference data scales with inference budget rather than labeler headcount.
• The recipe rests on weak-to-strong supervision. The supervisor does not need to exceed the model it trains, only encode the right signal on the dimensions that matter. This is what lets a model supervise a model.
• The AI labeler is only useful if it is calibrated against human judgment, because its biases are amplified through the optimization loop. Three failure modes appear when the constitution is the only signal: coverage gaps (unwritten dimensions go unconstrained), stylistic leakage (the labeler's own tone and format preferences enter the labels), and principle-application drift (interpretation of vague principles shifts across prompts).
• OpenAI's Model Spec is the productized form of a constitution that is detailed, hierarchical, versioned, and published. Its chain of command resolves conflicts between instructions at different authority levels (root, system, developer, user, guideline), addressing the policy pillar that principles alone handle weakly.
• A team can align its own model to a custom spec without a labeler workforce: draft the spec from observed production failures, distill it with the critique-and-revise loop (running SFT only, and skipping PPO, still captures most of the lift), and test the result against the spec.
• Model spec midtraining inserts a stage between pretraining and fine-tuning that trains the model on synthetic documents about the spec. Teaching the model the reasoning behind each clause lets it generalize the spec to prompt the demonstration data never covered.
• Deliberative alignment teaches the model how to reason over principles and specs using the chain of thought. This training carries over to inference time.
• Model spec midtraining inserts a stage between pretraining and fine-tuning that trains the model on synthetic documents about the spec. Teaching the model the reasoning behind each clause lets it generalize the spec to prompt the demonstration data never covered.
• Constitutional AI maps onto the three pillars of alignment: direct for principled alignment (the constitution is the principles), partial for policy alignment (encodable as principles, but with interpretive risk), and personality alignment inherits the AI critic's style priors.

FAQ