CUDA for LLMs

“A fundamental guide for CUDA and modern GPU architecture and their application in the field of AI.”
—Bryce Adelstein Lelbach, NVIDIA

The difference between a model that infers in milliseconds instead of seconds, or one that trains in days instead of weeks, is in how effectively you use your hardware. To squeeze every last teraflop out of an NVIDIA GPU, you need to write your own CUDA kernels, and CUDA for LLMs will get you there! This book is a friendly hands-on tour of the modern GPU programming stack, written for the latest NVIDIA hardware, and grounded in fundamentals that will outlast any single generation of chips. In 408 fully-illustrated, code-rich pages, author Elliot Arledge guides you from your first kernel all the way to Flash Attention, tensor cores, and multi-GPU distributed training.

GPU programming is complicated, so CUDA for LLMs builds your skills one layer at a time using a six-level kernel optimization stack. You’ll start by writing a vector addition kernel, learning how grids, blocks, and threads map onto real hardware. From there, you’ll implement the naive versions of the kernels every deep learning model depends on: GEMM, softmax, 1D and 2D convolution, max pooling, and matrix transpose. Then you’ll build and train a full neural network in six progressively faster implementations and dig into cuBLAS, with backpropagation derived and coded from scratch.

Throughout this book, your focus will stay on real optimization work. You’ll port a transformer to CUDA inference and attack each kernel in turn. Your GEMV decoder gets coalesced access, block-level reduction, and vectorization. Softmax becomes online, warp-shuffled, and fused. GEMM progresses through thread remapping, shared memory tiling, 1D and 2D blocktiling, and vectorized loads. Every optimization is profiled, and every profile is explained.

The final chapters take you to the frontier. You’ll program tensor cores using the WMMA and WGMMA instruction sets for Hopper hardware and see how manual tiling gets replaced by hardware-managed data movement. You’ll implement Flash Attention with tiling and online softmax, quantize models, and explore scaling up to 16 GPUs with tensor and pipeline parallelism over NCCL. By the end, you’ll have both a portfolio of more than 50 working kernels and the mental models to write new ones for hardware that doesn’t exist yet.