Harshwardhan Fartale

GPU Programming with Triton

Until recently, writing GPU kernels for LLM training and inference meant learning low-level programming tools like CUDA and C++. Triton, an open source, Python-based DSL created by OpenAI, bridges the gap between high-level machine learning frameworks and low-level GPU programming. Triton is built into PyTorch 2 and backed by NVIDIA, Intel, AMD, and Red Hat.

In GPU Programming with Triton, you'll learn how to work within the Triton ecosystem, from writing your first kernel to implementing advanced LLM features like FlashAttention and Native Sparse Attention. You'll use Triton to deliver the kernel-level control, fusion power, and acceleration that frameworks like PyTorch need under the hood without dropping down to CUDA and C++.

In this practical book written for readers with no previous GPU programming experience, author Harshwardhan Fartale introduces Triton’s innovative block-level programming model that replaces the tedious manipulation of low-level threads required by CUDA. Written for the latest hardware and LLMs, this book teaches GPU programming and Triton together, in Python, by profiling real workloads, identifying bottlenecks, and understanding why each optimization (coalescing, tiling, shared memory, reductions, and fusion) actually works.

As you go, you'll build the kernels that power modern AI systems, including FlashAttention, Native Sparse Attention, sparse matrix multiplication, and on-chip fused operations. You'll learn to profile real workloads, find the bottlenecks, wrap your kernel for production, and integrate it end to end into PyTorch. Each chapter includes handpicked practice problems designed to build the fluency that makes working in Triton feel like second nature.