Overview

1 Why parallel programming?

This chapter explains why parallel programming became necessary: for years, software got faster simply because hardware improved, but that “free lunch” ended when single-core processors hit limits imposed by heat and power. Since clock speeds could no longer keep rising safely, chip designers turned to multiple cores, especially GPUs with thousands of smaller processing units, to keep performance increasing.

The chapter contrasts sequential and parallel thinking by showing that some jobs, like doubling every element in an array, can be split into many independent tasks and run at once, while others, like summing values or computing Fibonacci numbers, contain dependencies that limit parallelism. It introduces the idea that the real question is not how many cores exist, but whether the work can be divided into parts that do not need one another’s results right away, and it shows how techniques such as reduction can still extract parallel speedup from partially dependent problems.

It also explains why GPUs and CUDA matter. GPUs were originally designed for graphics, where enormous amounts of similar work can be done independently, and CUDA made them accessible for general-purpose computing. The chapter emphasizes the host-device relationship between CPU and GPU, the difference between latency and throughput, and the kinds of problems that suit each chip: CPUs for complex, branching, low-latency work, and GPUs for large-scale, regular, data-parallel workloads such as matrix math, image processing, and AI.

Sequential processing: one core, one element at a time. A single CPU core works through a 1,000-element array step by step, doubling each value before moving on to the next. With only one core doing the work, the total number of steps equals the size of the data: 1,000 elements means 1,000 steps, and the time grows right along with it.
Sequential processing: one core, one element at a time. A single CPU core works through a 1,000-element array step by step, doubling each value before moving on to the next. With only one core doing the work, the total number of steps equals the size of the data: 1,000 elements means 1,000 steps, and the time grows right along with it.
Parallel processing: every element at once. A GPU assigns one thread per element, doubling all 1,000 values in parallel; the hardware schedules those 1,000 threads onto its many CUDA cores.
Parallel processing: every element at once. A GPU assigns one thread per element, doubling all 1,000 values in parallel; the hardware schedules those 1,000 threads onto its many CUDA cores.
Where the transistors go. A CPU devotes most of its silicon to control logic and cache, leaving room for only a few ALUs; a GPU gives a larger share of the same silicon to row upon row of small ALUs and connects to its DRAM through many more memory channels. The two designs are answers to opposite questions: finish one task fast? Or finish many tasks at once? The drawing is a schematic, not to scale; a real GPU still spends substantial silicon on control logic and cache, just a smaller share than a CPU does.
Where the transistors go. A CPU devotes most of its silicon to control logic and cache, leaving room for only a few ALUs; a GPU gives a larger share of the same silicon to row upon row of small ALUs and connects to its DRAM through many more memory channels. The two designs are answers to opposite questions: finish one task fast? Or finish many tasks at once? The drawing is a schematic, not to scale; a real GPU still spends substantial silicon on control logic and cache, just a smaller share than a CPU does.

Summary

  • Around 2005, heat killed the free lunch: single-core speed gains hit a physical ceiling.
  • Instead of one fast processor, use many working simultaneously. A modern laptop has 6–16 cores, high-end desktops well past 24; a GPU has thousands.
  • Parallel computing did not start with GPUs: MPI spreads work across separate machines, OpenMP across the cores of one CPU, and SIMD across values inside a single core. GPUs are a recent branch of that family, and they specialize in data parallelism: the same operation applied to many pieces of data at once.
  • How much of a problem can run at once depends on its dependencies more than on the core count. Before parallelizing anything, ask whether any task needs another task’s result before it can start.
  • Amdahl’s Law: the sequential parts of your code limit the overall speedup, no matter how many cores you throw at it. But Gustafson’s Law offers a brighter view: scale the problem size with the cores, and parallelism keeps paying off.
  • Thousands of simple processing units make GPUs well suited to massively parallel tasks like image processing, matrix math, and AI training.
  • CPUs and GPUs are partners: the CPU (the host) is built for low latency and runs your main program, while the GPU (the device) is built for high throughput and takes on the big parallel jobs.
  • A GPU pays off when a problem offers plenty of independent, identical work with regular memory access; small data, branch-heavy logic, and step-by-step dependencies favor the CPU.
  • Learning CUDA teaches you parallel thinking that carries over to just about any parallel framework, and helps you see what high-level libraries like PyTorch do underneath.
  • swforces.com, a free online lab, lets you write real CUDA code and check that its logic is correct, no GPU of your own required.
  • Part 2 builds up the recurring parallel patterns (Map, Reduce, Scan, Histogram, and Stencil), and Part 3 turns to the performance techniques that make them run fast on real hardware.

FAQ

What changed in programming when the “free lunch” ended?The “free lunch” was the era when new hardware made existing programs run faster without any code changes. It ended when single-core processors hit heat and power limits, so performance gains shifted from faster clocks to parallel hardware.
Why did CPU base clock speeds plateau around 3–4 GHz?They plateaued because higher clock speeds produced too much heat and power density for cooling systems to handle efficiently. Modern CPUs now throttle themselves when they get too hot.
Why did chip makers start adding more cores instead of making one core faster?Because making a single core much faster ran into physical limits. Adding more cores was a way to increase total work done, as long as the problem could be split into parallel tasks.
What is parallel computing?Parallel computing means using more than one processing unit at the same time to solve a single problem. It can happen on CPU cores, GPU cores, clusters of machines, or even within a single core using vector instructions.
What is the difference between data parallelism and task parallelism?Data parallelism runs the same operation across many pieces of data at once, like doubling every element in an array. Task parallelism runs different operations at the same time, like compressing a file while encrypting another.
Why are GPUs so much faster than CPUs for some jobs?GPUs are built for high throughput, meaning they finish many simple tasks at once. They contain thousands of small processing units, making them ideal for workloads with lots of independent, repetitive work.
What kinds of problems are good fits for a GPU?Problems with lots of independent, identical work and regular memory access patterns are good GPU candidates. Examples include image processing, matrix math, and other tasks where the same operation repeats over large data sets.
What is the host-device model in CUDA?In the host-device model, the CPU is the host that runs the main program, and the GPU is the device that does the parallel heavy lifting. The CPU sends work and data to the GPU, then receives results back.
Why is summing a list of numbers not embarrassingly parallel?Because each addition depends on the running total produced by the previous step. That dependency means the work cannot all happen independently, although it can still be sped up with parallel reduction.
Why should beginners learn CUDA instead of only using high-level libraries?Learning CUDA teaches how GPU programs really work, including memory movement, synchronization, and parallel patterns. That deeper understanding helps you diagnose performance problems and applies to other parallel frameworks too.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Grokking GPU Programming Grokking GPU Programming ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Grokking GPU Programming Grokking GPU Programming ebook for free