21 Asynchronous communication with a message queue
This chapter introduces asynchronous messaging with queues as a way to decouple components and improve flexibility compared to synchronous request/response calls. Instead of direct connections, publishers send messages to a queue and subscribers receive them when ready, which increases resilience to outages and spikes. Queues use lightweight, efficient protocols for high throughput, and Docker makes it easy to run the same enterprise-grade, open source queue technology in every environment. The discussion uses Redis and NATS to illustrate core concepts such as publishers, subscribers, and channels/subjects, and explains how acknowledgments, retries, and delivery semantics shape system behavior.
A hands-on refactor of a to-do app shows how queues unlock performance and scale. The web frontend publishes a “new item” event to NATS instead of writing directly to the database; initially nothing appears to happen because Redis and NATS drop messages when no subscribers exist, so a message handler is added to consume the event and persist data. The queue acts as a buffer, smoothing traffic bursts and letting a small pool of handlers process messages one at a time, effectively rate-limiting database connections while supporting high user load. The trade-off is eventual consistency—the UI may not show the new item immediately—suggesting event-driven UI updates or a manual refresh. Because handlers are small and stateless, they can be scaled horizontally, with the queue balancing work across instances.
Once events become the integration surface, new features can be delivered with zero downtime by deploying additional handlers that react to the same subject—such as an audit component—without changing the original app. Multiple publishers are also supported, exemplified by adding a separate REST API that emits the same event so different interfaces can evolve independently. The chapter surveys common patterns—publish–subscribe, request–response, fire-and-forget, and scatter–gather—and notes trade-offs across technologies: Redis and NATS excel at fast, in-memory delivery, while RabbitMQ adds durability and richer routing. Overall, Docker-hosted queues make event-driven architectures accessible, enabling systems that are scalable, resilient, and flexible to change.
Message queues decouple components so that they don’t communicate with each other directly.
A message queue is just like any other background container, waiting for connections.
The Redis CLI is a simple way to send messages to the queue running in the Redis container.
A subscriber to the queue receives a copy of every message published on the channel.
NATS, an alternative message queue, is lightweight and has an admin API.
The to-do app with messaging publishing; without any subscribers, there are missing features.
A simple subscriber that logs messages is a good to way to verify that they’re being published.
Asynchronous processing uses a message handler as well as an event publisher.
Message handlers subscribe to the queue, receive a copy of every message, and act on it.
Multiple message handlers share the workload so that you can scale to meet demand.
Publishing events decouples the components of your app and lets you add new features.
Events can have many subscribers and many publishers, which makes for loose coupling.
Pub-sub messaging lets many processes act on the same message being published.
Request-response messaging is client-service communication without a direct connection.
Lab
It’s the final lab of the book, and this one’s a little bit sneaky. The goal is to add another message handler for the to-do app—one that changes the text for items after they’ve been saved. That handler already exists, so this is mostly about wiring the new service into the Compose file, but there are some configuration settings you’ll need to dig around for too.
Your solution needs to run a new handler using the image diamol/ch21-mutating-handler:2e from Docker Hub, and when you have that working, there are a couple of things to investigate:
- The new component listens for events called events.todo.itemsaved, but nothing publishes those events yet. You’ll need to search for a config setting you can apply to one of the existing components to make it publish those events.
- The new component has a bad set of default configuration, so it’s not using the right address for the message queue. You’ll need to search for the setting and fix that too.
This is not as nasty as it seems; the answers you need are all in the Dockerfiles, and you’re just going to set values in your Compose file—no need to change source code or rebuild images. It’s a useful exercise because you’ll certainly spend some time trying to figure out config settings when you use Docker for real, and the final message handler adds a useful feature to the to-do app.
My solution is up on GitHub, as always, with a screen shot to prove it works: https://github.com/sixeyed/diamol/blob/2e/ch21/lab/README.md.
FAQ
What is asynchronous messaging, and how is it different from synchronous calls?
In synchronous calls (like REST or gRPC), a client holds an open connection and waits for the server’s response. Asynchronous messaging adds a queue between components: publishers send messages to the queue and return immediately; subscribers process messages when they’re ready. This decouples availability and timing—clients and services don’t need to be online or responsive at the same moment.Why use a message queue in containerized applications?
Queues decouple components, absorb load spikes, and use fast, lightweight protocols. They let you: handle many more short-lived connections than databases, buffer work during peaks, reduce client-side timeout/control logic, and scale publishers and subscribers independently. Docker makes running production-grade queues (like Redis and NATS) trivial across all environments.How do publishers, subscribers, channels, and subjects relate in Redis and NATS?
All app components are queue clients. A publisher sends messages; a subscriber receives them. Redis groups messages by channels (string names). NATS uses subjects (string topics) to classify message types. Subscribers listen on the specific channel/subject they care about, and receive messages published after they start listening.How does a queue improve the scalability of the to-do app’s save workflow?
Instead of the web app directly inserting into the database, it publishes “new item” events. A pool of message handlers subscribes and writes to the DB. The queue buffers bursts, handlers process one message at a time, and DB connections are bounded by the number of handlers—allowing many web instances without exhausting DB connections.What happens to messages when no subscribers are listening?
Behavior depends on the technology. Redis and NATS acknowledge receipt but drop messages if no subscribers are present; new subscribers only get messages published after they subscribe. Other systems may support dead-letter queues or persistence for later delivery.What is a message handler, and how does scaling handlers work?
A message handler is a subscriber that performs work for a specific message type (e.g., save new to-do items). You can run multiple handler instances; NATS distributes messages among them (round-robin) so they share the load. This increases throughput while keeping downstream usage (like DB connections) under control.What is eventual consistency, and how does it affect the user experience?
With async processing, the system becomes consistent after handlers finish processing messages. A user may briefly not see newly added items because the database update lags the UI redirect. Typical mitigations include optimistic UI updates, background refresh, or event-driven UI updates when “list changed” events arrive.How do message queues enable zero-downtime feature delivery?
Publish-subscribe decouples producers from consumers. You can add new handlers (e.g., an audit logger) that subscribe to existing subjects without changing or restarting the web app. The new container deploys alongside existing ones, immediately receiving a copy of relevant events.Which async messaging patterns are covered, and when should I use them?
- Publish-subscribe: Broadcast events to zero or more consumers (great for decoupled side effects and new features).- Request-response: Client sends a request message and waits for a response message (avoids direct connections while keeping a reply path).
- Fire-and-forget: Send a command without caring about a reply.
- Scatter-gather: Publish a request to many consumers and aggregate their responses.
When might I choose Redis, NATS, or RabbitMQ?
- Redis: Simple, fast pub-sub; great for ephemeral events where persistence isn’t required.- NATS: Cloud-native, very fast in-memory messaging with subjects and simple admin API; good for container-to-container comms and load sharing.
- RabbitMQ: Rich routing, durable queues, and persistence; choose it when you need message durability, complex routing, or advanced delivery guarantees.
Learn Docker in a Month of Lunches, Second Edition ebook for free
