3 Evaluating reasoning models

Evaluation is presented as the key to separating models that merely produce plausible-sounding text from those that actually solve problems correctly. The chapter centers on a verification-based approach for math tasks, where the model’s final answer is compared against a reference solution using deterministic tools rather than another language model. This kind of setup is especially valuable because it introduces verifiable rewards, a concept that will later support reinforcement learning for reasoning models.

The chapter then builds the verification pipeline step by step. It starts by loading a pre-trained Qwen3 base model, generating responses to math prompts, and wrapping the generation process in a convenient helper function. Next, it shows how to reliably extract the final boxed answer from the model’s output, even when formatting varies or the boxed expression is missing, and then normalize the extracted text into a consistent canonical form. To compare answers robustly, it uses SymPy so mathematically equivalent expressions can be recognized even if they look different syntactically, and it extends the checker to handle tuple-like answers by splitting them into parts before grading.

With these pieces in place, the chapter loads the MATH-500 benchmark and evaluates the model end to end. A prompt template is used to encourage boxed final answers, though the chapter also shows that prompt wording can strongly affect results. Running the full pipeline on a small subset demonstrates how predictions are generated, extracted, normalized, and graded automatically, while logging outputs for inspection. The base model performs modestly, but the reasoning variant does much better, illustrating both the benefits of reasoning-focused training and the cost of longer, more compute-intensive generations. Overall, the chapter delivers a reusable evaluation framework that will be reused later for comparing model improvements and training with verifiable rewards.

A mental model of the topics covered in this book. This chapter covers evaluation methods (stage 2), with a special focus on implementing verifiers, which we will later reuse as the basis for verifiable rewards in chapter 6.

Evaluating an LLM with a verification-based method in free-form question answering. The model generates a free-form answer (which may include multiple steps) and a final boxed answer, which is extracted and compared against the correct answer from the dataset.

A step-by-step workflow for building and applying a math verifier. Starting with a pre-trained LLM, we generate answers, extract and normalize them, and then compare them against the ground-truth solutions. Verified answers are then graded, and the process is repeated across a dataset (MATH-500) to evaluate overall model performance.

Rendered response with step-by-step calculations and the final boxed answer.

Illustration of steps 1 and 2 from the verifier workflow. A pre-trained LLM is loaded and prompted with a math problem, producing an output in raw LaTeX syntax. The answer is also shown in the rendered and more readable form.

An illustration of how the boxed result from the LLM output is extracted.

An illustration of how the boxed result from the LLM output is extracted and converted into a canonical plain form. This normalized answer is then later used for verification against the correct answer.

An illustration of how an LLM-generated answer is checked against the correct reference answer (ground truth). The final boxed answer is extracted and normalized, then compared to the correct answer provided in the dataset. If both match, the response is graded as correct.

Loading the evaluation dataset. After completing steps 2–6 on individual problems (generate, extract, normalize, verify, and grade answers) in the previous sections, the two remaining steps are to load the full dataset (step 7) and apply the same procedure across all problems to evaluate the model (step 8).

The complete evaluation pipeline on the MATH-500 dataset. After loading the dataset (step 7), steps 2–6 are applied systematically across all problems to obtain the final model evaluation (step 8).

Mental model of the topics covered in this book. This chapter implemented a verifier-based evaluation pipeline. In the next chapter, we will improve the reasoning capabilities of the LLM via more advanced inference techniques.

• There are four main evaluation methods for LLMs: multiple choice, verifiers, leaderboards, and LLM judges
• Verification-based evaluation methods allow free-form answers and use external tools to check correctness
• This chapter focuses on verification-based evaluation by building a math verifier that extracts, normalizes, and checks answers with SymPy
• The verification pipeline involves several core steps from loading the LLM to running the evaluation on a dataset
• As part of the verification pipeline, answer extraction uses string parsing to locate boxed content (with fallback mechanisms for missing boxes)
• Another step implements normalization, which standardizes diverse answer formats by stripping LaTeX and converting mathematical notation
• Finally, the pipeline uses mathematical equivalence checking (via SymPy) to compare expressions symbolically
• The MATH-500 dataset provides 500 curated math problems for evaluation
• Prompt templates significantly impact model performance
• The reasoning model achieves higher accuracy than the base model, but requires much longer runtime

FAQ