8 Prompt sampling

Prompt sampling treats each model reply as one probabilistic draw rather than a final truth, so reliability improves when you generate multiple independent responses and combine them intelligently. This chapter builds on earlier prompt-design methods by focusing on what to do when even a strong prompt can still produce different answers from run to run, especially when sampling settings like temperature introduce variation. The central idea is to match the aggregation method to the output type and the decision at stake.

For numerical outputs, the recommended technique is the mean, which reduces the impact of outliers and works well for scores, estimates, and confidence values. For categorical outputs, the mode is more appropriate because it functions as majority voting over labels such as severity or routing decisions. For full-text analytical responses, self-consistency checks whether independently generated drafts converge on the same conclusion, while best-of-N is used when one complete response must be selected as the strongest candidate under explicit criteria.

The chapter also gives practical guidance on how many samples to use, how to interpret spread or vote splits as confidence signals, and when sampling is worth the extra cost. Moderate temperature and a small number of generations are usually enough to create useful diversity without turning outputs into noise, and sampling is most justified when wrong answers are costly or when output quality varies enough to affect decisions. Even then, sampling is presented as a reliability aid rather than a substitute for human review, especially in production settings.