Appendix B. Exercise solutions
The exercise solutions focus on practical ways to adapt the book’s code for broader inputs, better evaluation, and more robust training behaviors. They show that tokenizers can handle unknown words by splitting them into smaller pieces, while familiar non-English words may remain intact, suggesting prior exposure in training. The solutions also emphasize that many code changes are small and localized, such as removing a hardcoded CPU device setting so the same notebooks can run on other hardware.
A major theme is improving evaluation on math tasks by testing more realistic cases and measuring outcomes more carefully. The solutions add edge-case tests for normalization and extraction, clarify when a test is poorly designed, and show how to compute average response length from evaluation logs without rerunning the model. They also report that longer reasoning-style outputs are much more verbose than base-model outputs, and that results on small subsets of MATH-500 can differ substantially from the full dataset, making larger or custom evaluation sets more informative.
Later solutions explore prompting and search strategies such as chain-of-thought, temperature and top-p sampling, self-consistency, Best-of-N selection, heuristic or logprob-based tie-breaking, and early stopping when a majority answer emerges. Across these methods, the reasoning model usually benefits more from structured inference than the base model, while prompt wording can strongly affect the base model but less so the reasoning model. The final chapters move into training and reward design, showing that format-aware rewards, conditional rewards, and removing think tokens each change stability and performance in different ways, generally reinforcing that careful reward shaping and dataset preparation matter more than superficial loss improvements.
Figure B.1 Basic metrics from a GRPO training run with a conditional format reward.
Figure B.2 Additional metrics from a GRPO training run with a conditional format reward.
Figure B.3 Distribution of training and validation set lengths.
FAQ
Where can I find the complete code examples for the exercise solutions?
The complete code examples are available in the supplementary GitHub repository: https://github.com/rasbt/reasoning-from-scratch.
How does the tokenizer handle unknown words and non-English text?
Unknown words are broken into smaller subword pieces or even single tokens, which lets the tokenizer handle any input. In contrast, the German words in the example were tokenized as whole words, suggesting the tokenizer had seen German text during training.
How can I rerun the chapter 2 code on a non-CPU device?
You can delete the line device = torch.device("cpu") in section 2.5 and rerun the rest of the chapter 2 code as is.
What interesting test cases can be added to the chapter 3 evaluation?
Examples include different bracket types like [1, 2] vs (1, 2), scientific notation like 1e-3 vs 0.001, algebraic simplification like (-3)^2 vs 9, and Unicode minus signs like −1 vs -1.
Why did the test with text around the answer fail in chapter 3?
Because run_demos_table is meant to test grade_answer, not to extract the answer from surrounding text. For text-based answers, you should first call extract_final_candidate and pass its result into the test.
How is the average response length on MATH-500 computed?
One way is to count the tokens in each generated answer and average them. In the example, the base model averaged about 97 tokens, while the reasoning model averaged about 892 to 1361 tokens depending on device, showing that reasoning responses are much longer.
How can chain-of-thought prompting be added to the MATH-500 evaluation?
By appending a suffix such as "\n\nExplain step by step." after the prompt template is rendered, with only a small change needed in the evaluation function.
How does self-consistency sampling break ties among answers?
If there is a tie, one simple rule is to choose the first answer in results["majority_winners"]. The book also shows how to use a heuristic scorer or average logprob scorer as a smarter tie-breaker.
What happens if all rollout rewards are identical in GRPO?
The advantages become all zeros after subtracting the mean, so the policy gradient is zero and no model update occurs for that prompt. This is intentional because there is no relative signal to learn from.
What is the effect of removing <think></think> tokens during distillation?
Omitting the think tokens slightly worsens results in most cases. The validation loss may look a bit better for one model, but that does not translate into better MATH-500 performance.
Build a Reasoning Model (From Scratch) ebook for free
