3 Colocation

Colocation is presented as the first data-centric latency optimization: place compute where the data and users are. Inter-node round trips over long distances impose tens to hundreds of milliseconds due to geography and last-mile effects, and even inside a single host, the memory hierarchy and core topology cause large latency swings (roughly: L1 ~1 ns, last-level cache ~10 ns, DRAM ~100 ns, disk microseconds, network milliseconds). The chapter’s guiding method is to optimize outside-in: minimize network distance and hops between clients and services, then tune within the machine by aligning threads, interrupts, and hot data with CPU caches and local memory.

At the inter-node layer, geographic and last‑mile latency set a hard floor, so avoiding long round trips is more impactful than micro-tuning. Practical strategies include multi-region deployment, near-edge points of presence and far-edge sites, and CDNs that cache and route requests nearby; modern programmable edges can also execute application logic. On the host, the OS network stack introduces latency via interrupts, protocol processing, copies, and context switches. Mitigations include pinning or co-locating interrupt handling with application threads, disabling small-message batching delays like Nagle’s when appropriate, choosing transports wisely (UDP, TCP, or reliability layers over UDP), and moving packet processing closer to hardware with kernel-bypass, NIC offload, or eBPF/XDP.

Intra-node colocation targets multicore architecture. With NUMA, keep hot data close to the cores that use it; partition state and steer requests so each shard is served by a local core/NUMA node (often with thread-per-core designs), and aim to fit working sets in the last-level cache to avoid DRAM stalls. A hands-on comparison underscores the payoff: a REST API that queries a local PostgreSQL process exhibits millisecond-scale, variable tails, whereas an embedded SQLite variant—eliminating network stack crossings and process boundaries—achieves microsecond medians and dramatically lower tail latency, shifting the whole distribution left and boosting throughput by orders of magnitude. The overarching lesson is clear: colocate compute and data first—across geography, processes, and cores—before chasing smaller optimizations.

An example of a distributed system. In the diagram, there are three different client nodes: two mobile phones and one desktop web browser. The clients interact with HTTP servers that are running on the cloud, for example. The HTTP servers interact with databases and other HTTP servers to perform client requests.

Geographic latency. The geographic distance between two nodes implies latency. For example, Hibernia Express advertises 60 ms round-trip between New York and London. Light tarvels the same distance in vacuum in 19 ms so the minimum theoretical round-trip latency is 38 ms.

Packets traversing the network stack.

UMA and NUMA architectures. The left side of the diagram represents a UMA architecture where all CPU cores have the same access latency to DRAM. The right side, on the other hand, represents a NUMA architecture with two nodes. CPUs that are in the same node can access NUMA node local memory faster than memory in a remote node.

REST API latency eCDF comparing SQLite and PostgreSQL. You can see how the tail latency of SQLite is around 15 milliseconds compared to 60 milliseconds with PostgreSQL. You can also see that SQLite latency is almost a straight vertical line, which indicates that much of the latency distribution is constant. One obvious explanation for this behavior is that SQLite version doesn’t have the variability in latency of network access.

• Colocation is a technique of bringing two components closer to reduce latency, which can be beneficial for applications accessing data in a database or communicating with other systems.
• Colocation is useful in distributed systems as it reduces geographic and last-mile latency. Content delivery networks (CDNs) and edge computing data and computation closer to the users, thereby reducing latency and improving the user experience.
• In multicore systems, memory topology is essential for optimizing low latency. Colocating compute and data by ensuring that a CPU core accesses memory on the local NUMA node is critical. Optimizing CPU caching and the network stack is also vital for reducing internal machine latency and improving the performance of latency-sensitive applications.
• One practical example of how to use colocation in a real-world system is a REST API server. You can reduce latency by orders of magnitude by deploying the REST API closer to the data, as we saw when we embedded SQLite into the REST API server.

FAQ