CUDA for LLMs

you own this product
Building fast transformer kernels
  • November 2026
  • ISBN 9781633434899
  • 408 pages
  • printed in black & white

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


CUDA for LLMs
Look inside
“A fundamental guide for CUDA and modern GPU architecture and their application in the field of AI.”
—Bryce Adelstein Lelbach, NVIDIA


The difference between a model that infers in milliseconds instead of seconds, or one that trains in days instead of weeks, is in how effectively you use your hardware. To squeeze every last teraflop out of an NVIDIA GPU, you need to write your own CUDA kernels, and CUDA for LLMs will get you there! This book is a friendly hands-on tour of the modern GPU programming stack, written for the latest NVIDIA hardware, and grounded in fundamentals that will outlast any single generation of chips. In 408 fully-illustrated, code-rich pages, author Elliot Arledge guides you from your first kernel all the way to Flash Attention, tensor cores, and multi-GPU distributed training.

GPU programming is complicated, so CUDA for LLMs builds your skills one layer at a time using a six-level kernel optimization stack. You’ll start by writing a vector addition kernel, learning how grids, blocks, and threads map onto real hardware. From there, you’ll implement the naive versions of the kernels every deep learning model depends on: GEMM, softmax, 1D and 2D convolution, max pooling, and matrix transpose. Then you’ll build and train a full neural network in six progressively faster implementations and dig into cuBLAS, with backpropagation derived and coded from scratch.

Throughout this book, your focus will stay on real optimization work. You’ll port a transformer to CUDA inference and attack each kernel in turn. Your GEMV decoder gets coalesced access, block-level reduction, and vectorization. Softmax becomes online, warp-shuffled, and fused. GEMM progresses through thread remapping, shared memory tiling, 1D and 2D blocktiling, and vectorized loads. Every optimization is profiled, and every profile is explained.

The final chapters take you to the frontier. You’ll program tensor cores using the WMMA and WGMMA instruction sets for Hopper hardware and see how manual tiling gets replaced by hardware-managed data movement. You’ll implement Flash Attention with tiling and online softmax, quantize models, and explore scaling up to 16 GPUs with tensor and pipeline parallelism over NCCL. By the end, you’ll have both a portfolio of more than 50 working kernels and the mental models to write new ones for hardware that doesn’t exist yet.

what's inside

  • CUDA and GPU programming from the ground up
  • Tensor and pipeline parallelism across multiple GPUs
  • PyTorch C++ extensions for shipping custom kernels

about the reader

For software and AI engineers comfortable with C/C++. No prior CUDA experience required.

about the author

Elliot Arledge is a GPU performance engineer and educator. Every kernel in this book comes from his workflow of writing, profiling, and verifying CUDA on both consumer and datacenter hardware. He created freeCodeCamp’s 12-hour CUDA course and consults on deep learning performance.

concepts explained in this book

  • matrix-matrix multiplication: Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
  • Tiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
  • Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
  • Matrix-vector multiplication is the linear algebra operation of multiplying a matrix by a compatible vector to produce a vector of row-wise dot products.
  • A reference implementation is a clear, correctness-oriented implementation that demonstrates intended behavior and provides a basis for validating other implementations.
  • Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
  • Layer normalization standardizes neural-network activations across specified feature dimensions independently for each example, often using learned scale and bias parameters.
  • Inference is the process of applying a trained model to input data to produce predictions or other outputs.

See what each chapter teaches

The perfect second book on GPU programming, targeted at high-performance AI kernel authoring. Modern AI runs on tensor cores and NVLINK—this book teaches both.

Jeff Hammond, NVIDIA Finland

An excellent PyTorch-first guide teaching you when and how to introduce CUDA libraries to optimize real deep learning models.

Mark Saroufim, co-founder of GPU MODE

Takes you from a naive kernel all the way to tensor cores, Flash Attention, and distributed training, and every step builds on the previous one. Recommended for anyone who wants to stop guessing why their model is slow.

Mesut Oezdil, CNCF HAMi Maintainer
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • CUDA for LLMs CUDA for LLMs ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • CUDA for LLMs CUDA for LLMs ebook for free