1 Why Parallel programming?

The chapter opens by showing why parallel programming became necessary: for decades, software benefited from the “free lunch” of faster hardware, where the same code ran faster each year without changes. That era ended when CPU clock speeds hit a power and heat ceiling, forcing chip makers to stop relying on a single faster core and instead add more cores. As a result, the central challenge in modern computing is no longer just making one processor faster, but using many processing units effectively.

Parallel computing answers that challenge by spreading work across multiple processors, from CPU cores to the thousands of simple cores in a GPU. The chapter emphasizes that more hardware only helps when a problem can actually be divided into independent tasks, and it distinguishes data parallelism from task parallelism. Simple examples such as doubling an array show how GPUs can turn many separate operations into one coordinated burst of work, while problems like summation require more structured techniques, such as parallel reduction, to expose usable concurrency.

Not every problem benefits equally, because dependencies create sequential bottlenecks that limit speedup. The chapter explains this with Amdahl’s Law and Gustafson’s Law, which help estimate how much faster a program can become and when parallelization is worth the effort. It also motivates CUDA by explaining why GPUs, originally designed for graphics, are now ideal for large, repetitive, data-heavy workloads like AI, and why learning CUDA teaches broader parallel-thinking skills that transfer to other systems and frameworks.

Sequential processing: one core, one element at a time. A single CPU core works through a 1,000-element array step by step, doubling each value before moving on to the next. With only one core doing the work, the total number of steps equals the size of the data: 1,000 elements means 1,000 steps, and the time grows right along with it.

Parallel processing: every element at once. A GPU assigns one thread per element, doubling all 1,000 values in parallel; the hardware schedules these thousands of threads onto its many CUDA cores.

Where the transistors go. A CPU devotes most of its silicon to control logic and cache, leaving room for only a few ALUs; a GPU gives a larger share of the same silicon to row upon row of small ALUs and connects to its DRAM through many more memory channels. The two designs are answers to opposite questions: finish one task fast? Or finish many tasks at once? The drawing is a schematic, not to scale; a real GPU still spends substantial silicon on control logic and cache, just a smaller share than a CPU does.

• Around 2005, heat killed the free lunch: single-core speed gains hit a physical ceiling.
• Instead of one fast processor, use many working simultaneously. A modern laptop has 6–16 cores, high-end desktops well past 24; a GPU has thousands.
• Parallel computing did not start with GPUs: MPI spreads work across separate machines, OpenMP across the cores of one CPU, and SIMD across values inside a single core. GPUs are the newest branch of that family, and they specialize in data parallelism: the same operation applied to many pieces of data at once.
• Dependencies, not core count, decide how much of a problem can run at once. Before parallelizing anything, ask whether any task needs another task’s result before it can start.
• Amdahl’s Law: the sequential parts of your code limit the overall speedup, no matter how many cores you throw at it. But Gustafson’s Law offers a brighter view: scale the problem size with the cores, and parallelism keeps paying off.
• Thousands of simple processing units make GPUs well suited to massively parallel tasks like image processing, matrix math, and AI training.
• CPUs and GPUs are partners: the CPU (the host) is built for low latency and runs your main program, while the GPU (the device) is built for high throughput and takes on the big parallel jobs.
• A GPU pays off when a problem offers plenty of independent, identical work with regular memory access; small data, branch-heavy logic, and step-by-step dependencies favor the CPU. Match the tool to the task.
• Learning CUDA teaches you parallel thinking that carries over to just about any parallel framework, including PyTorch.
• swforces.com, a free online lab, lets you write real CUDA code and check that its logic is correct, no GPU of your own required.
• Part 2 builds up the recurring parallel patterns (Map, Reduce, Scan, Histogram, and Stencil), and Part 3 turns to the performance techniques that make them run fast on real hardware.

FAQ