9 Exploiting concurrency

This chapter explains how exploiting concurrency is a primary lever for reducing latency, especially in systems that perform long-running or I/O-bound work. It distinguishes concurrency (many tasks making progress) from parallelism (tasks executing simultaneously), and uses Little’s Law to show that keeping throughput high at high concurrency is essential to avoid latency inflation from resource contention. Practical motivation comes from services like REST APIs, where overlapping I/O and computation prevents slow operations from stalling overall responsiveness.

The chapter surveys concurrency models and their trade-offs for low latency. Threads are widely supported but incur kernel-level context-switch overheads and cache thrashing; they shine for parallelism but can be costly for fine-grained concurrency. User-space options such as fibers (cooperative scheduling) and coroutines (async/await with futures/promises) reduce blocking and overhead, while event-driven designs revolve around a non-blocking event loop. The actor model isolates state and coordinates via asynchronous message passing, simplifying shared-state hazards. Beyond concurrency, the chapter covers parallel processing strategies: data parallelism (partitioning and locality, SIMD/GPUs) and task parallelism (independent operations, processor affinity and NUMA awareness), emphasizing that careful partitioning and placement boost throughput while acknowledging coordination and Amdahl-law limits.

The chapter then examines the consistency–concurrency trade-off in transactional systems. It contrasts serializability and strict serializability with snapshot isolation (via MVCC), highlighting anomalies like write skew and the implications of weaker isolation levels (e.g., dirty/fuzzy reads and lost updates). It outlines pessimistic control with two-phase locking versus optimistic MVCC, noting MVCC’s concurrency benefits and storage/GC costs. A micro-benchmark ties the themes together: running independent 10 ms tasks sequentially versus concurrently shows an order-of-magnitude latency reduction, underscoring the core guidance—choose an appropriate concurrency model, apply data and task parallelism thoughtfully, and select isolation/concurrency control that meets correctness needs without sacrificing latency.

Concurrency vs parallelism. Concurrent execution (on the left) interleaves thread execution on a single core, which provides an illusion of work happening at the same time because all threads make some progress quickly. In contrast, parallel execution (on the right) executes the threads simultaneously on different cores, providing true parallel execution.

The OS runs multiple threads at the same time (but not simultaneously) on a single CPU core by context switching between them. In this example, every thread is run for exactly 4 milliseconds at a time and then the OS switches to another thread. In real-world systems, a thread might block to wait for I/O, for example, causing the OS to switch to another thread more quickly.

Event loop is the heart of event-based concurrency. The event loop polls for events from the OS and invokes appropriate event handlers. For example, if a packet arrives from the network, the event loop polls for them using an I/O polling API such as io_uring, epoll, or kqueue, and then hands over the received messages to event handlers. When the event loop polling is done, the event loop calls itself to continue polling.

Summing values of an array sequentially means stepping through each element at a time. However, with data parallelism, you can reduce latency by summing subsets of the array in parallel.

An example of key-value store with two serializable transaction that update the database. The first transaction updates objects A and B, which are both atomically committed. The second transaction updates object A twice with only the second update being visible to other transactions.

Example of multi-version concurrency control (MVCC) across two transactions. The steps are the same as in Figure 9.5 on serializability. However, in the second transaction, we see how the database maintains multiple versions of the objects, which allows transactions to execute optimistically.

• Concurrency models enable multiple tasks to execute simultaneously to reduce latency, though true parallelism specifically refers to performing tasks simultaneously across multiple processors - each approach requires different design considerations for optimal performance.
• Threading provides a widely-used concurrency model but comes with kernel-level context switching overhead, while coroutines offer lightweight user-space managed concurrency through suspendable execution, and event-driven architectures handle concurrency through event loops and non-blocking operations.
• Parallel execution can reduce your application latency in contrast to sequential execution because you can potentially complete more work in less time. Data parallelism improves performance by executing identical operations simultaneously across different data portions (especially effective with SIMD and GPUs), while task parallelism runs different independent tasks simultaneously - these complementary approaches can be combined for maximum performance gains.
• Transaction isolation levels represent different tradeoffs between consistency guarantees and performance, from read uncommitted to serializable, with snapshot isolation providing a practical middle ground through multi-version concurrency control (MVCC).
• Concurrency control mechanisms like MVCC allow databases to execute transactions concurrently while maintaining consistency, helping reduce latency compared to serial execution while preserving critical correctness guarantees for applications.

FAQ