5 Inference-time scaling via self-refinement

This chapter explains self-refinement as an inference-time scaling technique for improving LLM reasoning without additional training. Unlike self-consistency, which generates multiple independent answers and selects one by majority vote, self-refinement starts with one answer and asks the model to critique and revise it. The chapter frames this as a trade-off: spending more inference compute can improve accuracy, but it also increases latency and cost. It also emphasizes that shorter or higher-confidence answers are not automatically better; evaluating responses requires scoring methods that capture useful properties such as extractable final answers, brevity, and model-assigned likelihood.

The chapter first builds scoring tools that can be used to compare candidate answers and decide whether a refined answer should replace an earlier one. A simple heuristic scorer rewards answers with clearly extractable final results, especially boxed answers, and adds a mild brevity bonus so concise responses are preferred when quality is otherwise similar. The chapter then develops a probability-based scorer by explaining next-token probabilities, sequence probabilities, and why raw products of probabilities become numerically unstable for longer outputs. To address this, it introduces log-probabilities, where multiplying probabilities becomes summing log-probabilities. The resulting average log-probability scorer evaluates only the answer tokens, excludes the prompt, and normalizes by answer length so responses of different sizes can be compared more fairly.

The chapter then implements self-refinement by having the model generate an initial response, critique that response, and produce a revised answer based on the critique. This loop can be repeated for multiple iterations, optionally using either the heuristic scorer or the average log-probability scorer to accept a revision only when its score improves. In examples, self-refinement can correct wrong answers, but benchmark results show that its effectiveness depends strongly on the model and scorer. For the base model, self-refinement improves accuracy moderately but remains less effective than self-consistency on the tested math task; for the reasoning model, self-refinement with heuristic scoring gives stronger gains, while average log-probability scoring can select fluent but incorrect answers. The chapter concludes that self-refinement is useful but not a universal fix, and it prepares the ground for later training-based methods where log-probabilities become central again.

A mental model of the topics covered in this book. This chapter continues stage 3 and focuses on inference-time techniques for improving reasoning without additional training. This chapter introduces self-refinement, where the model iteratively critiques and improves its own answers.

Three inference-time methods to improve reasoning covered in this book. The first two methods were covered in the previous chapter. This chapter covers the third method, self-refinement, where the model iteratively improves its own answers.

We build a simple rule-based score, compute token probabilities and log-probabilities, and then use these scores as part of a self-refinement method where the model iteratively improves its own answers.

This section implements a rule-based scorer to rank different answers generated by the pre-trained LLM.

Two generated responses reach the same correct answer but differ in their explanations. A scorer evaluates the responses and assigns a score to each response.

A simple rule-based length penalty used by our scorer. Longer explanations receive a smaller score contribution.

In this section, we move from a simple rule-based scorer to token-level probabilities. These probabilities form the basis for the logprob scoring method that we will use later in the self-refinement approach.

How an LLM selects the next token. The model converts the input text into token IDs, computes a score for every vocabulary token, and chooses the token with the highest score. The plot on the right shows the logit values for a subset of the vocabulary, with the token for Berlin having the highest score.

How we look up the logit scores for specific tokens. After passing the input text through the model, we obtain a logit value for every vocabulary token. We then convert the candidate tokens we want to score into token IDs and read off their corresponding logit values from the distribution.

Next-token scoring. The input text is converted into token IDs and fed to the LLM, which outputs logits for the next token. After applying a softmax, these logits become probabilities, where tokens like "Berlin" receive high probability and unlikely alternatives such as "Bridge" receive values near zero.

Computing token probability scores for a given sequence. For each position, we feed the preceding text into the model and read off the softmax probability of the next token. Multiplying these conditional probabilities yields the joint probability of the full sequence.

Extracting next-token probabilities. After converting the input text into token IDs, the model computes logits that are transformed into probabilities with a softmax function. Using index tensors for positions and true next tokens, we then get the model's computed probability for each next token.

Overview of how we move from token probabilities to token log-probabilities, which provide a numerically more stable basis for log-probability scoring used later in self-refinement.

Comparison of logits, softmax probabilities, and log-probabilities for a simple example. The log-probabilities preserve the ordering of the probabilities.

How logits are converted to probabilities and log-probabilities for next-token scoring. The correct next token ("Berlin") receives a high logit, which becomes a high probability and a less negative log-probability, while unlikely candidates like "Bridge" map to very small probabilities and large negative log-probabilities.

How token-level log-probabilities accumulate to form sequence log-probabilities. Each row shows the log-probability of the next token given the preceding text. Summing these values results in the joint log-probability of the full sequence

This section implements a logprob scorer, based on token log-probabilities, which we will use in the self-refinement method later in this chapter.

The modified logprob scoring procedure. The prompt tokens are excluded from the calculation, and only the log-probabilities of the answer tokens are collected. These values are then averaged to obtain a length-normalized score, which allows us to compare answers of different lengths fairly.

The final step in our workflow, where the logprob scorer developed earlier is used inside the self-refinement method.

The self-refinement process. The LLM first produces an initial answer to the prompt, then receives a critique prompt that asks it to analyze its own response and produce a short critique with a plan to refine the answer. In the final step, the model is given a refine prompt that contains the original question, its draft answer, and the critique, and it generates a revised answer that incorporates the suggested improvements.

The self-refinement loop with optional scoring. The model first produces an initial answer, then critiques it, and generates a revised answer based on the critique. Both answers can be evaluated with the scoring functions from this chapter (for example, logprob scoring), and the revised answer is only accepted if its score improves on the previous one.

Overview of the book's progression from basic LLM usage to inference-time reasoning methods and finally to training-based techniques. This chapter concludes the inference-scaling methods without additional training, and the next chapters introduce approaches that update model weights to further improve reasoning performance.

• Self-refinement extends the inference-time scaling ideas from the previous chapter by iteratively critiquing and improving a single answer instead of relying on multiple independent samples as in self-consistency.
• A simple rule-based scoring function ranks model outputs by rewarding extractable final answers and shorter, more economical completions.
• Next-token scoring quantifies model confidence by converting logits into normalized probabilities and combining these into a sequence-level likelihood.
• Log-probabilities replace raw probabilities to avoid numerical underflow and to turn products over many tokens into stable sums and averages.
• These scoring functions are more generally useful beyond self-refinement, for example, for breaking ties in self-consistency or implementing Best-of-N selection strategies.
• The self-refinement procedure consists of three stages: generating an initial draft, producing a short critique and fix plan, and generating a revised answer.
• A reusable refinement function automates the self-refinement workflow with multiple iterations (refinement rounds), and it can use a score-based acceptance to keep only revisions that do not degrade a computed answer score using one of the scoring functions we developed earlier.

FAQ