1 When PyTorch just isn’t enough
Large language models demand enormous amounts of compute, and the difference between slow training or inference and practical performance often comes from how well the GPU is used. This chapter frames CUDA as the low-level layer beneath PyTorch on NVIDIA hardware, explaining why learning it matters when built-in framework operations are not fast enough, not flexible enough, or not tailored to a specific workload. It emphasizes that although higher-level tools can be powerful, understanding raw CUDA gives you direct control over what the GPU actually executes.
The chapter introduces the core CUDA mental model: a CPU host coordinates work, while the GPU device carries out massively parallel computation in its own memory space. CUDA kernels are functions launched across thousands of threads, with each thread handling one small part of the problem. The key insight is to recognize data-parallel patterns, where the same operation is applied independently to many values, as in activations, convolutions, attention, and gradient updates. It also explains that performance depends heavily on the GPU memory hierarchy, where global memory is large but slow, shared memory and L1 are faster, and registers are fastest, so efficient programs keep data as close to the compute units as possible.
The chapter also sets expectations for optimization: first kernels will usually be slower than PyTorch, but the book will show how to climb a ladder of improvements through better memory access, reduced traffic, tensor cores, kernel fusion, quantization, and eventually multi-GPU scaling. It highlights when GPUs help most and when CPU execution may still be better, such as with tiny problems, low arithmetic intensity, or irregular branching. By the end, the reader is prepared to think like a performance engineer, understand why custom kernels can beat standard libraries in specific cases, and build a foundation for writing and optimizing their own high-performance GPU code.
A CPU dedicates more area to caches and control logic for a few powerful cores; a GPU dedicates more area to many simpler arithmetic cores.
The four-step CUDA workflow: allocate GPU memory, copy data to the device, execute the kernel, and copy results back.
Write code for one thread; CUDA launches thousands of parallel copies, each with a unique thread index.
Element-wise ReLU: each matrix element is processed by a dedicated thread running the same operation.
Memory hierarchy on a GPU
The optimization ladder shows multipliers at each level of optimization covered throughout the book. These represent the compound effects on efficiency and performance from latency, memory savings, occupancy, throughput, and cost savings.
Summary
- GPUs excel at throughput by running the same instruction across massive data sets in parallel, making them ideal for deep learning workloads.
- Host code (CPU) orchestrates work while device code (GPU) executes it. Every CUDA project involves explicit memory transfers between CPU RAM and GPU memory.
- CUDA launches thousands of threads organized into grids and blocks. Each thread computes a unique index to determine which data element it processes.
- Matrix multiplications, convolutions, activations, and attention computations all fit the “same operation, different data” pattern. Recognizing these opportunities is the first skill of a CUDA performance engineer.
- Fast kernels respect the GPU memory hierarchy: registers and shared memory are scarce but fast, and global memory is plentiful but slower. The key optimization strategy is to keep data in the fastest memory tier for as long as possible.
- The rest of this book climbs an optimization ladder, from writing your first CUDA program in chapter 2 to applying advanced memory and computation techniques in later chapters.
FAQ
What is CUDA, and how is it related to PyTorch on NVIDIA GPUs?
CUDA is NVIDIA’s programming model for writing code that runs directly on the GPU. PyTorch uses CUDA under the hood when you move a model to a GPU, for example withmodel.to("cuda"). In other words, PyTorch often hides the low-level details, but CUDA is the layer that actually executes GPU work.
CUDA for LLMs ebook for free