Appendix E. Batching and throughput-oriented execution
This appendix explains why batching is not used throughout the main examples by default: processing one prompt at a time keeps the code simpler, easier to debug, and often more practical on limited hardware. Batching is presented as an optional extension for situations where modern GPUs and enough memory are available, especially when the goal is to process many problems, sample multiple responses, or train on multiple examples at once.
The key distinction is between latency and throughput. Single-example generation is best when you want the quickest answer for one prompt or a straightforward conceptual implementation, while batching is a throughput optimization that can dramatically reduce total runtime for large workloads. However, batching is not a universal speedup, because padding, masking, and other bookkeeping add overhead and can cancel out gains on CPUs or smaller or less optimized devices.
To make batching work, the appendix shows how different-length prompts are packed into rectangular tensors using padding, usually with left padding, and how attention masks preserve the behavior of ordinary autoregressive generation by preventing padded positions from influencing results. It then applies this idea across the book’s supplementary scripts: batched evaluation for MATH-500, batched self-consistency sampling, batched GRPO rollouts, and batched distillation. The overall takeaway is that batching often improves runtime substantially, but it usually costs extra memory and requires careful handling of padding and masks, so the best batch size depends on the hardware and workload.
Figure E.1 An overview of single-example versus batched generation. In batched generation, several prompts are packed into one batch, processed in parallel, and then decoded together to improve throughput.
Figure E.2 Sequential generation (1) versus regular batched generation (2) and early-stop batched generation (3). The regular batched helper (2) keeps finished rows in the batch as placeholder EOS rows, whereas the _stop variant (3) removes finished rows from the active compute batch as soon as they emit EOS.
Figure E.3 The causal mask used in single-example generation. Tokens can attend only to themselves and earlier positions, which is the standard autoregressive setup.
Figure E.4 Left padding in batched generation. The shorter row is padded up to the maximum sequence length in the batch, and the padding-aware mask ensures that these padded positions do not affect attention.
FAQ
What is the main difference between single-example generation and batched generation?Batched generation processes several prompts or examples together in one parallel compute step, while single-example generation handles one prompt at a time. Batched execution is mainly used to improve throughput, whereas single-example execution is simpler and often better for debugging and minimizing latency.Why does batching improve throughput?Batching lets the model process multiple inputs in parallel, which can reduce total runtime when the hardware supports it well. This is especially useful on modern GPUs with enough memory, where multiple prompts, samples, or training examples can be evaluated at once.What is the difference between latency and throughput?Latency is how quickly the model returns an answer for one prompt. Throughput is how many prompts or examples the system can process in a fixed amount of time. Batching mainly targets throughput rather than latency.Why is batching not always faster?Batched execution adds overhead such as padding, masking, and extra bookkeeping. On CPUs or less optimized GPUs, especially with small models, those overheads can outweigh the benefits of parallelism, so batching may provide little or no speedup.Why do batched prompts need padding?Prompts often have different token lengths, but PyTorch tensors must be rectangular. Padding extends shorter sequences so all rows in the batch have the same length, allowing them to be packed into one tensor.What is the purpose of the attention mask in batched generation?The attention mask marks which positions are real tokens and which are padding. It ensures padded positions do not affect attention, so batched execution behaves more like single-example execution.Why does the appendix use left padding for batched prompts?Left padding aligns shorter sequences with the rightmost, actual prompt tokens so that all prompts in the batch share the same final length. This makes it easier to apply padding-aware causal masking during generation.What is the difference between generate_text_basic_batched_cache and generate_text_basic_batched_cache_stop?The regular batched helper keeps finished rows in the batch by forcing them to keep emitting EOS tokens until the longest row ends. The _stop variant removes finished rows from the active compute batch as soon as they emit EOS, which is more efficient.When do chapters 4 and 6 use batching without padding-aware masking?Chapters 4 and 6 batch repeated copies of the same prompt or rollout setup. Since every row starts with the same length, they do not need padding-aware batching and can use the regular Qwen3Model implementation.What are the practical trade-offs of using larger batch sizes?Larger batch sizes can significantly improve throughput and reduce total runtime, but they also increase memory usage. In practice, you should start with a smaller batch size and increase it until you reach the best balance for your hardware and memory limits.
pro $24.99 per month
access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
Build a Reasoning Model (From Scratch) ebook for free
