RabbitMQ in Depth
Gavin M. Roy
  • MEAP began June 2013
  • Publication in Spring 2016 (estimated)
  • ISBN 9781617291005
  • 375 pages (estimated)
  • printed in black & white

RabbitMQ in Depth is a practical guide to building and maintaining message-based systems. This book covers detailed architectural and operational use of RabbitMQ with an emphasis on not just how it works but why it works the way it does. You'll find examples and detailed explanations of everything from low-level communication to integration with third-party systems. You'll also find the insights you need to make core architectural choices and develop procedures for effective operational management.

Table of Contents show full


1. Foundational RabbitMQ

1.1. RabbitMQ’s features and benefits

1.1.1. RabbitMQ and Erlang

1.1.2. RabbitMQ and AMQP

1.2. Who is using RabbitMQ, and how?

1.3. The advantages of loosely coupled architectures

1.3.1. Decoupling your application

1.3.2. Decoupling database writes

1.3.3. Seamlessly adding new functionality

1.3.4. Replication of Data and Events

1.3.5. Multi-Master Federation of data and events

1.4. The Advanced Message Queuing Model

1.5. Summary

2. How to speak Rabbit: The AMQ protocol

2.1. AMQP as an RPC transport

2.1.1. Kicking off the conversation

2.1.2. Tuning in to the right channel

2.2. AMQP’s RPC Frame Structure

2.2.1. AMQP frame components

2.2.2. Types of frames

2.2.3. Marshaling messages into frames

2.2.4. The anatomy of a method frame

2.2.5. The content header frame

2.2.6. The body frame

2.3. Putting the protocol to use

2.3.1. Declaring an exchange

2.3.2. Declaring a queue

2.3.3. Binding a queue to an exchange

2.3.4. Publishing a Message to RabbitMQ

2.3.5. Consuming messages from RabbitMQ

2.4. Writing a message publisher in Python

2.5. Getting messages from RabbitMQ

2.6. Summary

3. An in-depth tour of message properties

3.1. Using properties properly

3.1.1. Creating an explicit message contract with content-type

3.1.2. Reducing message size with gzip and content-encoding

3.1.3. Referencing messages with message-id and correlation-id

3.1.4. Born-on dating: The timestamp property

3.1.5. Automatically expiring messages

3.1.6. Balancing speed with safety using delivery-mode

3.1.7. Validating message origin with app-id and user-id

3.1.8. Getting specific with the message type property

3.1.9. Using reply-to for dynamic workflows

3.1.10. Custom properties using the headers property

3.1.11. Avoiding the priority property

3.1.12. A property you can’t use: cluster-id/reserved

3.2. Summary

4. Performance tradeoffs in publishing

4.1. Balancing delivery speed with guaranteed delivery

4.1.1. What to expect with no guarantees

4.1.2. RabbitMQ won’t accept non-routable messages with mandatory set

4.1.3. Publisher Confirms as a lightweight alternative to transactions

4.1.4. Using alternate-exchanges for unroutable messages

4.1.5. Batch processing with transactions

4.1.6. Surviving node failures with HA queues

4.1.7. HA queues with transactions

4.1.8. Persisting messages to disk via delivery-mode 2

4.2. When rabbits push back: TCP Backpressure

4.2.1. Checking the Client Status with rabbitpy

4.2.2. Using the Management API to detect backpressure

4.3. Summary

5. Don’t Get Messages; Consume Them

5.1. Basic.Get vs Basic.Consume

5.1.1. Basic.Get

5.1.2. Basic.Consume

5.2. Performance tuning consumers

5.2.1. Using No-Ack Mode for Faster Throughput

5.2.2. Controlling consumer prefetching via quality of service settings

5.2.3. Using Transactions with Consumers

5.3. Rejecting messages

5.3.1. Basic.Reject

5.3.2. Basic.Nack

5.3.3. Dead Letter Exchanges

5.4. Controlling queues

5.4.1. Temporary queues

5.4.2. Permanent queues

5.4.3. Arbitrary queue settings

5.5. Summary

6. Message Patterns via Exchange Routing

6.1. Simple message routing using the Direct exchange

6.1.1. Creating the application architecture

6.1.2. Creating the RPC Worker

6.1.3. Writing a simple RPC publisher

6.2. Broadcasting messages via the Fanout exchange

6.2.1. Modifying the facial detection consumer

6.2.2. Creating a simple image hashing consumer

6.3. Selectively routing messages with the Topic exchange

6.4. Selective routing with the Headers exchange

6.5. Exchange performance benchmarking

6.6. Going Meta: Exchange-to-exchange routing

6.7. Routing messages with the Consistent-Hashing exchange

6.8. Summary


7. Scaling RabbitMQ with Clusters

7.1. About clusters

7.1.1. Clusters and the Management UI

7.1.2. Cluster node types

7.1.3. Clusters and queue behavior

7.2. Cluster setup

7.2.1. Virtual Machine setup

7.2.2. Adding nodes to the cluster

7.3. Summary

8. Cross-Cluster Message Distribution

8.1. Federating exchanges and queues

8.1.1. Federated Exchanges

8.1.2. Federated Queues

8.2. Creating the RabbitMQ Virtual Machines

8.2.1. Creating the first instance

8.2.2. Duplicating the EC2 instance

8.3. Connecting upstream

8.3.1. Defining federation upstreams

8.3.2. Defining a policy

8.3.3. Leveraging upstream sets

8.3.4. Bi-directional federated exchanges

8.3.5. Federation for cluster upgrades

8.4. Summary

9. Keeping a watchful eye on RabbitMQ servers


10. Using Alternative Protocols

10.1. MQTT and RabbitMQ

10.1.1. The MQTT Protocol

10.1.2. Publishing via MQTT

10.1.3. MQTT Subscribers

10.1.4. MQTT Plugin Configuration

10.2. STOMP and RabbitMQ

10.2.1. The STOMP Protocol

10.2.2. Publishing Messages

10.2.3. Consuming Messages

10.2.4. Configuring the STOMP Plugin

10.2.5. Using STOMP in the web browser

10.3. Stateless Publishing via HTTP

10.3.1. How statelessd came to be

10.3.2. Using statelessd

10.3.3. Operational architecture

10.3.4. Publishing Messages via statelessd

10.4. Summary

11. Big data and RabbitMQ

12. Database Integrations

12.1. The pg_amqp PostgreSQL Extension

12.1.1. Installing the pg_amqp extension

12.1.2. Configuring the pg_amqp extension

12.1.3. Publishing a Message via pg_amqp

12.2. Listening to PostgreSQL Notifications

12.2.1. Installing the PostgreSQL LISTEN Exchange

12.2.2. Policy Based Configuration

12.2.3. Creating the Exchange

12.2.4. Creating and Binding a Test Queue

12.2.5. Publishing via NOTIFY

12.3. Storing Messages in InfluxDB

12.3.1. InfluxDB Installation and Setup

12.3.2. Installation of the InfluxDB Storage Exchange

12.3.3. Creating a Test Exchange

12.3.4. Testing the Exchange

12.4. Summary

13. Writing RabbitMQ plugins

13.1. Setting Up Your Development Environment

13.1.1. Getting the Public Umbrella

13.1.2. Building RabbitMQ

13.2. Creating a Wrapper Plugin

13.3. Creating the Redis Storage Exchange

13.3.1. Project Structure and Source

13.3.2. Specifying the Application Details

13.3.3. Creating the Exchange Module

13.3.4. Creating the Policy Validation Module

13.3.5. Creating the gen_server Worker Module

13.3.6. Supervising the Worker

13.3.7. Implementing the Support Library

13.4. Compiling and Running the Redis Storage Exchange

13.4.1. Building and Enabling the Exchange

13.4.2. Testing the Exchange

13.5. Summary

Appendix A: Getting Set Up

A.1. Installing VirtualBox

A.2. Installing Vagrant

A.3. Setting up the Vagrant Virtual Machine

A.4. Confirming your installation

A.5. Summary

About the Technology

Any large application needs an efficient way to handle the constant messages passing between components in the system. Billed as "messaging that just works," the RabbitMQ message broker initially appeals to developers because it's lightweight, easy to set up, and low maintenance. They stick with it, though, because it's powerful, fast, and up to nearly anything you can throw at it. This book takes you beyond the basics and explores the challenges of clustering and distributing messages across enterprise-level data-centers using RabbitMQ.

What's inside

  • Understanding the AMQP model
  • Communicating via MQTT, Stomp, and HTTP
  • Valuable troubleshooting techniques
  • Integrating with Java technologies like Hadoop and Esper
  • Database integrations with PostgreSQL and Riak

About the reader

Written for programmers with a basic understanding of messaging oriented systems and RabbitMQ.

About the author

Gavin M. Roy is an active open-source evangelist and advocate who has been working with Internet and Enterprise technologies since the mid-90's. As the CTO of MeetMe.com, Gavin implemented RabbitMQ as the core of MeetMe's event-based scalable application architecture. Gavin is a member of the Python Software Foundation and is the maintainer of pika, a pure-Python implementation of the AMQP protocol for RabbitMQ.

Manning Early Access Program (MEAP) Read chapters as they are written, get the finished eBook as soon as it’s ready, and receive the pBook long before it's in bookstores.
MEAP combo $59.99 pBook + eBook
MEAP eBook $47.99 pdf + ePub + kindle

FREE domestic shipping on three or more pBooks