Why do decoupled and scalable systems need a message broker?

This question guides this entire article, where we explore RabbitMQ, one of the most popular and mature message brokers in the open-source ecosystem. We will explore the fundamental concepts, the benefits that RabbitMQ offers, and in practice, show, through examples and code, how to send and consume messages.

Introduction

Monolithic systems require all modules to “talk” directly to each other, creating strong dependencies and fragility. However, as applications grow and become increasingly distributed, the need to decouple components and services increases. This is why systems based on microservices architecture are growing.

In this type of architecture, a message broker acts as an intermediary between systems or microservices (producers and consumers): it receives messages from those who publish events and messages, and guarantees their delivery to those who consume and process them, in an asynchronous manner, allowing different parts of an application or separate systems to exchange information efficiently, even when they are not available at the same time.

In addition, it helps solve scalability and reliability problems and allows the integration of services developed in different languages.

The RabbitMQ architecture is simple: producers send messages to components called exchanges, which act as routers. The exchanges are configured with specific rules that determine which queues the messages should be sent to. Each queue, in turn, is consumed by one or more consumers, who read and process the messages according to their business logic.

RabbitMQ implements the AMQP (Advanced Message Queuing Protocol) protocol, is written in Erlang and adopts a multithreaded architecture, which allows high concurrency.

The management interface makes the tool attractive for simple maintenance and monitoring of queues, messages and connections.

Usage Examples

One of the main practical applications of RabbitMQ is in e-commerce order processing. It is also widely used in system integration in various sectors, such as logistics, transport, and billing. It allows heterogeneous systems to share data reliably and in an organized manner, even in highly complex scenarios.

Imagine an online shopping scenario. When an order is placed, a message is sent to a distribution queue, where the service responsible for stock consumes it and confirms the product’s availability. Another message can be sent to the transport queue, where the logistics system organizes delivery. Simultaneously, a message is processed in the billing queue, generating the invoice. This efficient and asynchronous organization avoids delays and promotes greater flexibility in communication between systems. The separation into queues ensures that each service works independently and scalably.

Asynchronous Nature

RabbitMQ uses an asynchronous pattern, allowing messages to be processed independently of the consumer’s execution time. This means that producers can publish messages without having to wait for consumers to process them immediately. This approach decouples systems, ensuring that the data flow continues to flow even when consumers are temporarily unavailable or dealing with high workloads.

This operating model is ideal for distributed systems, allowing scalability, flexibility, and resilience in message processing. With the ability to organize and prioritize message traffic, RabbitMQ becomes an assertive solution for implementing asynchronous architectures in various types of applications.

Benefits of RabbitMQ

Decoupling between components and systems: the producer does not directly know the consumer. This facilitates the maintenance and evolution of the system, as new consumers can be added without changing who produces the messages.

Resilience and fault tolerance: messages can be stored persistently. If a consumer fails, the messages remain in the queue until it or another consumer processes them.

Horizontal scalability: it is possible to add new workers to process messages as demand increases.

Integration of heterogeneous systems: RabbitMQ supports several languages (Python, Java, Ruby, Go, etc.), making it ideal for microservices architectures.

Reliability and performance: delivery confirmation, acknowledgment control, durable queues, and high throughput contribute to making systems robust.

Flexibility: different types of exchanges (direct, topic, fanout, headers) allow for varied routing patterns.

Monitoring and management: the web administration interface provides graphs and statistics of messages, connections, channels, exchanges, and queues, facilitating operations.

Fundamental Concepts

This section covers the fundamental concepts of RabbitMQ and messaging in general.

Message

RabbitMQ is a message-queueing software, meaning a message is the information to be queued and sent to the consumer service. A message can include any type of information. For example, it can contain information about a process or task that must be started in another application (which may even be on another server), a JSON with some system entity data, or it can just be a simple text message.

Producer

In the RabbitMQ context, a Producer is the system or application responsible for creating and sending messages to the broker in an exchange. It acts as the starting point in the messaging flow, generating information that needs to be processed or shared by other systems or services.

Producers do not interact directly with the queues where messages are stored. Instead, they send messages to the exchanges, which have the function of routing them to the appropriate queues based on the configured routing rules.

In an e-commerce application, for example, the system responsible for registering new orders acts as a producer, sending messages to an exchange. These messages may contain order information, such as the ID, purchased items, and customer data. The exchange then routes these messages to specific queues, where they will be consumed by services, such as stock control, billing, and logistics, for example.

The producer’s role is fundamental, as it initiates the communication process, ensuring that messages are delivered to RabbitMQ for later processing by consumers. This structure allows producers to focus only on creating and sending messages, while RabbitMQ manages the entire messaging flow.

Consumer

Consumers are the systems or applications responsible for consuming messages from queues. They act as the recipients of messages sent by producers and stored by the broker, processing the data according to the necessary business logic.

Consumers connect to specific queues and read the available messages asynchronously, or on demand, depending on the configuration. This flexibility allows the consumer to process messages at their own pace, ensuring that the system works efficiently even in high-load scenarios.

For example, in an e-commerce application, a consumer can be responsible for reading messages from an order queue and, based on this information, performing tasks such as updating inventory, generating an invoice, or triggering the shipping system.

Consumers transform the messages stored in the queues into concrete actions, allowing RabbitMQ to integrate various systems and ensure fluid communication between them.

It is possible to have multiple consumers competing for messages from the same queue.

Queue

A Queue is the location where messages are temporarily stored until they are consumed. It works as a buffer that ensures that the data sent by producers remains available to consumers, even if they are not ready to process it immediately.

Queues are organized sequentially, ensuring that messages are delivered in the order they were received. In addition, they can be configured with different properties, such as durability (Time-To-Live - TTL), to ensure that messages remain stored even in the event of a system restart, and auto-delete when there are no consumers.

Using e-commerce as an example again, messages related to new orders can be stored in a specific queue until they are consumed by stock control or payment processing services. This structure allows each consumer to process messages at their own pace, without directly depending on the producers’ execution time.

Exchange

It is the entity responsible for routing messages to one or more queues. The type of exchange determines how the message will be routed.

The direction of messages by an Exchange is essential to organize the message flow in distributed systems, allowing different types of data or events to be directed to the appropriate services.

For example, in a logistics application, an exchange can receive messages from a producer that reports package shipping events and status updates. Based on the configured rules, the exchange can route them to different queues: one queue for the transport service and another for the tracking system.

There are several types of exchanges:

Direct exchange: Routing based on an exact routing key. Useful for delivering a message to a specific queue.

Topic exchange: Uses wildcards in the routing key to match patterns. The character * represents one word and # represents zero or more words.

Fanout exchange: Ignores the routing key and sends the message to all queues bound to the exchange. Excellent for disseminating real-time events.

Headers exchange: Routing based on attributes defined in the message headers, not on the routing key.

Routing keys and bindings

The routing key is a string that the producer sends along with the message. The exchange uses this key to decide which queue to deliver the message to. A binding connects a queue to an exchange and defines the routing rule, for example: the logs.error queue can be associated with a log.error routing key. In a topic exchange, you can use log.* to receive all logs of a certain type and log.# to receive all logs.

In other words:

Binding: Bindings are rules that exchanges use (among other things) to route messages to queues.

Routing key: A key that the exchange uses to decide how to route the message to the queues. Think of the routing key as an address for the message.

Users

It is possible to connect to RabbitMQ with a specific username and password. Users can have associated permissions such as read, write, and configuration privileges within the instance (read, write, and configure). Users can also be given permissions for specific virtual hosts.

For production environments, delete the default user (guest)!.

The default user can only connect from localhost, as it has known credentials. Instead of enabling remote connections, consider creating a separate user with administrative permissions and a strong password. It is recommended to use a separate user per application. For example, if you have a mobile application, a web application, and a data aggregation system, create 3 separate users. This facilitates several things:

• Correlate client connections with applications.
• Use fine-grained permissions.
• Credential renewal - roll-over (for example, periodically or in case of a security breach).

Vhost

RabbitMQ is a multi-tenant system: connections, exchanges, queues, bindings, user permissions, policies and other elements belong to virtual hosts (vhosts), logical groups of entities.

Virtual hosts provide logical grouping and separation of resources. Users can have different permissions for different vhosts, and queues and exchanges can be created to exist only in one vhost.

A virtual host has a name. When a client connects to RabbitMQ, it specifies a vhost name to connect to. If authentication is successful and the provided username has permissions for the vhost, the connection will be established.

Communication Patterns

In addition to the simple work queue pattern, where several instances process tasks concurrently, there are also other patterns:

Pub/Sub (publish/subscribe): with the fanout exchange, the same message is delivered to all consumers, useful for notification and broadcast events.

Work queues: when several instances consume from the same queue, improving task distribution.

RPC (request/reply): a client sends a message requesting a result and waits for the response in the queue indicated in the reply_to header. The correlation is done by a correlation_id.

Installation and Configuration

The simplest way to run RabbitMQ locally is by using Docker. Follow these steps to provision a local RabbitMQ container (run in a terminal with Docker installed):

# 1. Download the official image with the management plugin
$  docker pull rabbitmq:3.13-management

# 2. Start the container with the default ports (5672 for AMQP and 15672 for the web interface)
$  docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management

# 3. Check the logs (optional)
$  docker logs -f rabbitmq

After a few seconds, the interface will be available at http://localhost:15672.

The default login is guest/guest. On the administration screen, you can monitor the volume of messages, connections, and server resources.

You can also create new users and configure permissions.

Optional: creating users and queues via script

If you want to automate the creation of queues and users, or avoid using the graphical interface, you can use commands via the HTTP API, as in the examples below:

# Create a user
$  curl  -u  guest:guest  -XPUT  -H  "content-type:application/json"  \
-d '{"password":"admin123","tags":"administrator"}' \
http://localhost:15672/api/users/myadmin

# Create a queue
$  curl  -u  myadmin:admin123  -XPUT  -H  "content-type:application/json"  \
-d '{"durable":true}' \
http://localhost:15672/api/queues/%2F/myqueue

Exploring the Management Interface

The RabbitMQ web interface provides graphs of message rates, queue message statistics (ready, unacknowledged), active connections, open channels, and information about each cluster node, such as memory usage, disks, and processes. The queued messages and message rate graphs are examples of these graphical statistics.

After starting RabbitMQ, we can monitor and manage RabbitMQ from the web interface at port 15672. You can access this page via the following URL: http://localhost:15672/ with the default username and password as guest/guest.

Tabs such as Connections, Channels, Exchanges, Queues and Streams and Admin allow you to manage each resource. The Queues tab, for example, shows all queues, allowing you to configure TTL, view pending messages, and even send messages manually.

Overview

The Overview page displays two graphs: one for Queued Messages and one for the message rate. You can change the period displayed on the graph by pressing the "Last Minute" tab above the graph. Information about all the different message states can be found by pressing the ? tab/button.

Creating a Producer and a Consumer in Python

In this part of the article, we will create two Python programs: a producer that sends messages, and a consumer that receives the messages and prints them to the screen. They are implemented using the pika library for Python. First, make sure you install the library with pip install pika.

For more details and examples in Python, start by accessing the examples in tutorial-one-python and the rabbitmq-tutorials repository.

Producer

In a directory, create a file for the producer RabbitMqProduce.py.

import pika

# Connection settings
def  connect():
return pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))

# Queue declaration and message sending
def  publish_message(body: str):
connection = connect()
channel = connection.channel()

# Ensures that the queue exists. "durable=True" persists the queue to disk
channel.queue_declare(queue='hello', durable=True)

# Publishes the message to the default exchange (""), using the routing_key equal to the queue name
channel.basic_publish(
exchange='',
routing_key='hello',
body=body,
properties=pika.BasicProperties(delivery_mode=2) # 2 = persistent
)

print(f"[x] Sent: {body}")
connection.close()

if  __name__ == '__main__':
publish_message('Hello, RabbitMQ!')

Consumer

In the same directory, create a file for the consumer RabbitMqReceive.py.

import pika

def  connect():
return pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))

# Callback called when a message arrives
def  callback(ch, method, properties, body):
print(f"[x] Received: {body.decode()}")

# Simulates processing
# When finished, sends ack (acknowledgment of processing)
ch.basic_ack(delivery_tag=method.delivery_tag)

# Consumption loop
def  start_consumer():
connection = connect()
channel = connection.channel()

# Declares the same queue with durable=True to ensure it exists and is persistent
channel.queue_declare(queue='hello', durable=True)

# Qos with prefetch=1 distributes only one message at a time to each worker
channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue='hello', on_message_callback=callback)

print(' [*] Waiting for messages. To exit press CTRL+C')

channel.start_consuming()

if  __name__ == '__main__':
start_consumer()

Topic example with topic exchange

The topic pattern allows flexible routing based on patterns. Imagine we have several queues interested in different log levels. We can create a topic type exchange called logs and associate queues with specific bindings:

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declares a topic type exchange
channel.exchange_declare(exchange='logs', exchange_type='topic', durable=True)

# Declares queue for error logs and associates it with the exchange with binding key "log.error"
channel.queue_declare(queue='error')
channel.queue_bind(exchange='logs', queue='error', routing_key='log.error')

# Declares queue for logs of all levels (with wildcard)
channel.queue_declare(queue='all')
channel.queue_bind(exchange='logs', queue='all', routing_key='log.#')

# Publishes messages
channel.basic_publish(exchange='logs', routing_key='log.error', body='A critical error occurred!')
channel.basic_publish(exchange='logs', routing_key='log.info', body='Important information...')

print('Messages sent.')

connection.close()

In the consumer, just associate the desired queue and process. The error queue will only receive messages with log.error, while the all queue will receive all (log.#). Note that the # character captures zero or more words in the routing key.

Creating a Producer and a Consumer in Java

If your preferred language is Java, this is an example of how to create the same programs: a producer that sends messages, and a consumer that receives the messages and prints them to the screen. They are implemented using the amqp-client library.

First, make sure to install the library in your project. If you use Maven, add to your pom.xml file:

<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>5.26.0</version>
</dependency>

For more details and examples in Java, start by accessing tutorial-one-java and the rabbitmq-tutorials/java repository.

Producer

In your project, create a producer class RabbitMqProduce.java.

package com.altabuild.rabbitmq;

import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;

import java.nio.charset.StandardCharsets;

public class RabbitMqProduce {
private final static String QUEUE_NAME = "hello";

public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
try (Connection connection = factory.newConnection();
Channel channel = connection.createChannel()) {
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
String message = "Hello World!";
channel.basicPublish("", QUEUE_NAME, null, message.getBytes(StandardCharsets.UTF_8));
System.out.println(" [x] Sent '" + message + "'");
}
}
}

Consumer

In your project, create a producer class RabbitMqReceive.java.

package com.altabuild.rabbitmq;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DeliverCallback;
import java.nio.charset.StandardCharsets;

public class RabbitMqReceive {

private final static String QUEUE_NAME = "hello";

public static void main(String[] argv) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();

channel.queueDeclare(QUEUE_NAME, false, false, false, null);
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");

DeliverCallback deliverCallback = (consumerTag, delivery) -> {
String message = new String(delivery.getBody(), StandardCharsets.UTF_8);
System.out.println(" [x] Received '" + message + "'");
};
channel.basicConsume(QUEUE_NAME, true, deliverCallback, consumerTag -> { });
}
}

Test Results

In both examples, Java or Python, the behavior of the applications and the result are the same.

First, run the consumer application to start the listening loop. This application keeps running and consuming any message that is posted to the hello queue.

Then, run the producer application. The producer application will create a message with the content [x] Sent: Hello, RabbitMQ!. Each message sent will be printed to the consumer’s console as soon as it is consumed.

The delivery_mode=2 parameter marks the message as persistent, ensuring that it is saved to disk and not just in memory.

It is possible to monitor the hello queue metrics through the RabbitMQ console to monitor performance.

For example, when a message is produced in the queue, it is possible to monitor the number of messages ready for processing:

When consuming messages from a queue, it is possible to monitor the dequeuing of messages in Queued messages:

That’s all, folks!

If you’ve made it this far, you’ve learned how a message broker is essential for anyone developing distributed systems and microservices, and the concepts involved in this important piece of messaging. RabbitMQ provides robustness, monitoring, and flexibility for complex architectures. Run RabbitMQ locally, and run the example codes yourself to see its use in practice.

Want to learn even more? I leave you with the following challenges as next steps:

• Run the producer made in Python, and the consumer made in Java (or vice versa) to see in practice how completely different systems can be integrated through queues.
• Explore the management interface, and check the configurations and models explained at https://www.rabbitmq.com/tutorials.
• Implement other patterns such as work queues and RPC.
• Test scalability by adding multiple consumers and understanding the behavior when multiple applications consume.

Mastering these concepts will be a differentiator to ensure that your applications are prepared for high volumes of messages and efficiently integrated.

Did you like the introduction? Then try integrating RabbitMQ into a real project to have more experience! The code and step-by-step presented here are just the beginning to explore the full potential of this powerful tool.