1 Why parallel programming?
This chapter explains why parallel programming became necessary: for years, software got faster simply because hardware improved, but that “free lunch” ended when single-core processors hit limits imposed by heat and power. Since clock speeds could no longer keep rising safely, chip designers turned to multiple cores, especially GPUs with thousands of smaller processing units, to keep performance increasing.
The chapter contrasts sequential and parallel thinking by showing that some jobs, like doubling every element in an array, can be split into many independent tasks and run at once, while others, like summing values or computing Fibonacci numbers, contain dependencies that limit parallelism. It introduces the idea that the real question is not how many cores exist, but whether the work can be divided into parts that do not need one another’s results right away, and it shows how techniques such as reduction can still extract parallel speedup from partially dependent problems.
It also explains why GPUs and CUDA matter. GPUs were originally designed for graphics, where enormous amounts of similar work can be done independently, and CUDA made them accessible for general-purpose computing. The chapter emphasizes the host-device relationship between CPU and GPU, the difference between latency and throughput, and the kinds of problems that suit each chip: CPUs for complex, branching, low-latency work, and GPUs for large-scale, regular, data-parallel workloads such as matrix math, image processing, and AI.
Sequential processing: one core, one element at a time. A single CPU core works through a 1,000-element array step by step, doubling each value before moving on to the next. With only one core doing the work, the total number of steps equals the size of the data: 1,000 elements means 1,000 steps, and the time grows right along with it.
Parallel processing: every element at once. A GPU assigns one thread per element, doubling all 1,000 values in parallel; the hardware schedules those 1,000 threads onto its many CUDA cores.
Where the transistors go. A CPU devotes most of its silicon to control logic and cache, leaving room for only a few ALUs; a GPU gives a larger share of the same silicon to row upon row of small ALUs and connects to its DRAM through many more memory channels. The two designs are answers to opposite questions: finish one task fast? Or finish many tasks at once? The drawing is a schematic, not to scale; a real GPU still spends substantial silicon on control logic and cache, just a smaller share than a CPU does.
Summary
- Around 2005, heat killed the free lunch: single-core speed gains hit a physical ceiling.
- Instead of one fast processor, use many working simultaneously. A modern laptop has 6–16 cores, high-end desktops well past 24; a GPU has thousands.
- Parallel computing did not start with GPUs: MPI spreads work across separate machines, OpenMP across the cores of one CPU, and SIMD across values inside a single core. GPUs are a recent branch of that family, and they specialize in data parallelism: the same operation applied to many pieces of data at once.
- How much of a problem can run at once depends on its dependencies more than on the core count. Before parallelizing anything, ask whether any task needs another task’s result before it can start.
- Amdahl’s Law: the sequential parts of your code limit the overall speedup, no matter how many cores you throw at it. But Gustafson’s Law offers a brighter view: scale the problem size with the cores, and parallelism keeps paying off.
- Thousands of simple processing units make GPUs well suited to massively parallel tasks like image processing, matrix math, and AI training.
- CPUs and GPUs are partners: the CPU (the host) is built for low latency and runs your main program, while the GPU (the device) is built for high throughput and takes on the big parallel jobs.
- A GPU pays off when a problem offers plenty of independent, identical work with regular memory access; small data, branch-heavy logic, and step-by-step dependencies favor the CPU.
- Learning CUDA teaches you parallel thinking that carries over to just about any parallel framework, and helps you see what high-level libraries like PyTorch do underneath.
- swforces.com, a free online lab, lets you write real CUDA code and check that its logic is correct, no GPU of your own required.
- Part 2 builds up the recurring parallel patterns (Map, Reduce, Scan, Histogram, and Stencil), and Part 3 turns to the performance techniques that make them run fast on real hardware.
Grokking GPU Programming ebook for free