1 Evaluations and alignments for AI
Evaluation and alignment are presented as the two core mechanisms for making AI systems useful, safe, and dependable in practice. Evaluation asks what a model or system actually does, while alignment asks what it should do, and together they form a feedback loop that guides development from early model versions to full production systems. The chapter emphasizes that success in AI engineering depends on measuring behavior carefully, identifying gaps, and then applying corrections that bring the system closer to intended goals.
The text explains that evaluating LLMs is harder than testing traditional software because their outputs are open-ended, probabilistic, and often subjective. To organize this complexity, it distinguishes verifiable tasks, where correctness can be checked objectively, from open-ended tasks, where multiple answers may be valid and judgment depends on context. It then surveys evaluation methods across this spectrum, including reference-based metrics that compare outputs to gold answers, reference-free methods that judge intrinsic quality, human evaluation, and LLM-as-a-judge approaches, while also highlighting practical issues such as answer extraction, verifier reliability, test coverage gaps, and hallucination detection.
On the alignment side, the chapter identifies three pillars: principled alignment, which embeds values like helpfulness and harmlessness; policy-based alignment, which enforces concrete rules and instruction hierarchies; and personality alignment, which shapes tone, style, and character consistency. It concludes by framing AI engineering as an iterative cycle of defining success, evaluating, analyzing errors, and aligning the system again, with evaluation treated as the reward signal that drives improvement. The overall message is to work backward from desired outcomes, use evidence rather than intuition, and build evaluation infrastructures that can reliably support production-ready AI systems.
A taxonomy of AI evaluation methods, distinguishing between verifiable tasks and open-ended tasks.
AI’s answer is correct but requires parsing. The model produces a chain-of-thought (COT) response where the final answer must be extracted from the surrounding text.
Open-ended evaluation tasks require navigating a landscape of subjective quality judgements and are broadly classified into Reference-based and reference-free evaluations.
Reference-based evaluation workflow. Human experts create reference answers from a test set. The AI system generates candidate responses from the same inputs. The evaluator then compares each candidate's response with the corresponding reference response, producing scores.
Reference-free evaluation workflow. Quality signals are defined for the task. The AI generates outputs for test inputs. Metrics assess each output’s intrinsic properties without reference answers. Scores reveal quality patterns and failure modes.
Example task evaluation instructions provided to Amazon Mechanical Turk data labeler.
Training progression showing how models learn format before knowledge. Source: “Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability” by Chang et al, 2023.
Example of hallucinated authorship in an AI response.
Three pillars of alignment and how they relate to each other.
The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.
Summary
- Evaluation measures whether a model, agent, or AI system performs the intended task accurately, reliably, and across edge cases. Alignment ensures the model’s behavior matches human goals, intentions, and values. Together, they ensure AI systems follow user intent and complete tasks safely.
- AI systems span a spectrum from verifiable to open-ended tasks.
- Verifiable tasks have objectively checkable outputs—coding and math are common examples.
- Open-ended tasks lack objective verification; examples include translation, report generation, and creative writing.
- There are two broad approaches to evaluating open-ended tasks, both aiming to approximate human judgment as closely as possible.
- Reference-based evaluations compare model outputs to expert-produced gold standards.
- Reference-free evaluations rely on intrinsic properties of generated content, sometimes guided by expert criteria.
- Evaluations may use heuristic, statistical, semantic, or LLM-as-a-Judge methods to align with human evaluators, even though humans often disagree.
- Hallucinations occur because AI systems learn structure and form before learning factual grounding—they master how to sound correct before knowing what is correct. Hallucination can be measured using faithfulness metrics.
- The three pillars of alignment are principled, policy-based, and personality-based.
- Principled alignment aligns AI with foundational human values.
- Policy-based alignment aligns AI behavior with specific rules, norms, or institutional policies.
- Personality-based alignment shapes consistent traits and communication styles in AI behavior.
- Evaluation and alignment are interdependent. Evaluation metrics become optimization targets, creating feedback loops that can lead to emergent or unintended behaviors.
- Working backward from desired outcomes distinguishes AI engineering from research and other forms of engineering — we define metrics first, then build systems to optimize them.
- There is no one-size-fits-all solution. Every evaluation and alignment technique involves trade-offs that must be carefully balanced for specific use cases.
LLM Evaluation and Alignment, The Foundational Ideas ebook for free
