7 Eliminating work

This chapter reframes low-latency engineering as a relentless search for work you can avoid. It urges an iterative, measurement-first approach: profile to find the real bottlenecks, then remove them—and don’t introduce new ones. The core levers span algorithmic choices, data movement and serialization, memory behavior, operating-system interactions, and strategic precomputation. The guiding principle is simple but powerful: the fastest code is the code you never execute.

Eliminating work starts with algorithmic complexity and data-structure selection: move from quadratic or exponential behaviors toward linear, logarithmic, or constant-time paths, and choose structures (hash tables, trees, queues) that fit your access patterns and latency targets. Reduce data transformation by minimizing serialization/deserialization and copies—prefer efficient formats (for example, binary) or even zero/low-copy approaches where feasible. Memory is a major source of non-determinism: dynamic allocation brings fragmentation and synchronization costs; garbage collection adds pauses (albeit tunable); and virtual memory’s demand paging, TLB behavior, and page size choices can swing latency. Hardware locality matters too—optimize for cache fit and NUMA by partitioning data, pinning threads, and binding memory to the right nodes.

The operating system adds its own overheads: scheduling delays, context switches, background daemons, and interrupts can all perturb tail latency. Techniques include limiting threads, raising priority, setting CPU affinity, adopting thread-per-core designs, isolating background work, using polling judiciously, steering NIC flows, or even bypassing the kernel network stack. When computation remains expensive or uncontrollable, precompute results (e.g., materialized views) and serve lookups instead. The chapter closes with a measurement workflow—benchmarking and flame-graph profiling—and a concrete example (Fibonacci) showing how moving from exponential recursion to linear memoization and then to an iterative, allocation-free version yields dramatic speedups. The overarching lesson: measure, rethink the algorithm, cut copies and allocations, tame the OS, and precompute—systematically removing work from the critical path.

Illustration of algorithmic complexity in Big O notation for constant time 𝑂(1), logarithmic time 𝑂(𝑙𝑜𝑔𝑛), linear time 𝑂(𝑛), and quadratic time 𝑂(𝑛2).

Serialization transforms in-memory representation (shown on the left as a Rust struct) to a serialized format such as JSON (shown on the right). Deserialization performs the opposite transformation from serialized format to in-memory representation.

The dynamic memory allocator manages a section of the memory we call the heap. In this example, the dynamic memory allocator maintains a list of free blocks in the memory (freelist), which is a linked list of all the available blocks.

Process virtual memory address space consists of sections that serve a different purpose. The text, data, and BSS sections are initialized at program startup, whereas the heap, memory-mapped region, and stack sections change dynamically at runtime (denoted with dotted line).

A page table is a data structure the CPU uses to translate virtual memory addresses into physical addresses. A page in virtual memory does not always have a counterpart in physical memory. Instead, the operating system can store a page on the disk, for example, and lazily move it to physical memory when it’s accessed.

A flame graph of the performance benchmark run, which visualizes how much time is spent on each function proportional to the total runtime.

• Eliminating work in low-latency application is critical as we move from data to code because sometimes the only way to reduce latency is to simply do less.
• Choosing algorithms and data structures with low algorithmic complexity is a key to reduce latency. In latency sensitive code, quadratic time is already pretty high because runtime increases so quickly as input size increases. Often times, to optimize for low latency, look for linear time or better algorithmic complexity.
• Serialization and deserialization in applications is a major source of latency, which you can often eliminate. For example, JSON is a popular text format for representing data, but has a high CPU and memory overhead from converting to and from in-memory representation. As an alternative, the Protocol Buffers framework provides a much more efficient serialization format, which can reduce request latency. Of course, even Protocol Buffers has some overhead, which can be eliminated by using an in-memory based representation such as FlatBufers.
• Various aspects of memory management, including dynamic memory allocation, garbage collection, virtual and physical memory management, has impact on latency. For example, dynamic memory allocation requires work for memory allocation but also for freeing because of freelist management. Similarly, although memory allocation with garbage collection is often fast, reclaiming back memory can introduce significant latency spikes. When working with low latency applications, avoiding memory allocation with static allocation or object pooling is an effective technique.
• Operating system has bunch of overheads, including the effects of scheduling delays, context switching, background tasks, and network stack management, which need to be tackled for low latency. Many low-latency applications end up either allocating resources such as CPU cores or memory for OS services to avoid interference with application treads or by-pass the OS with thread-per-core model and kernel-bypass networking.
• Precomputation can minimize runtime computations and improve response times in latency-sensitive applications by essentially calculating the results before they’re needed. Materialized views in databases are one typical example of precomputation in action, but come with the downside of increased storage requirements.
• Measuring application performance methodologically with benchmarks is the key to finding places to eliminate work. Using FlameGraph-based profiling and a benchmarking framework such as Criterion in Rust helps you establish sound performance testing to know what to optimize and how much.

FAQ