Best pub/sub messaging brokers

One of the most fundamental techniques for computers in a network to interact with one another is request/response or request/reply. It is widely utilized by HTTP, which is the World Wide Web’s basis for data transfer. HTTP follows the traditional client-server architecture, with a client initiating a connection to make a request, then waiting for a response.

While this client-server architecture is incredibly helpful and has many applications, it does have certain limits. For example, when communication is asynchronous or includes numerous nodes, the request/reply pattern becomes more difficult.

As a result, developers need an architecture pattern that can function asynchronously and scale well independently of the number of nodes. In this tutorial, we’ll explore the pub/sub architectural pattern, as well as the best message brokers for implementing it. Let’s get started!

Table of contents

• Introducing the pub/sub architecture
  • Publishers
  • Event bus/ broker
  • Subscribers
  • Topic-based filtering
  • Content-based filtering
• Pub/sub architecture use cases
  • Internet of Things
  • Event notifications
  • Data streaming
• Pub/sub messaging brokers
  • Apache Kafka
  • Active MQ
  • Redis
  • ZeroMQ, ØMQ or 0MQ
• Conclusion

Introducing the pub/sub architecture

Pub/sub, an abbreviation for publish/subscribe, is an asynchronous messaging architecture in which messages are transferred between entities without the sender or the recipient knowing the identity of the other.

In other words, the pub/sub architecture’s two sides are loosely connected, making it an appropriate design for integrating the nodes of an expanding distributed system. Pub/sub allows machines to interact and respond to data updates in real-time, which differs from the usual request/response communications paradigm in which data is updated at regular intervals.

There are three main components that make up the pub/sub architecture, publishers, the event bus or broker, and the subscribers. Let’s define these.

Publishers

Publishers are nodes that generate messages that are subsequently sent across the system using the event bus/broker.

Event bus/broker

The event bus/broker nodes serve as middlemen, facilitating the flow of messages from publishers to subscribers. Brokers further strengthen the decoupling between system nodes because subscribers deal with the broker rather than the entire system.

Subscribers

Subscribers essentially listen for communication about the topics and categories in which they are interested, doing so without knowing who the senders of these communications are.

Subscribers typically indicate their interest in receiving specific messages, acting as a filtering mechanism. Because of the fine-grained topic control, it’s simple to ensure that the various event buses are sending the correct message. We can achieve this type of filtering using either topic-based filtering or content-based filtering.

Topic-based filtering

Topic­ based filtering requires the messages to be disseminated into logical channels. The subscribers only get messages from logic channels to which they have subscribed.

Content-based filtering

Content-based filtering enables subscribers to receive messages based on the substance of the messages. The subscribers have filters that check whether the content of the messages being broadcast matches the constraints defined by the subscriber.

Pub/sub architecture use cases

The pub/sub architecture is a great option for distributed architectures with a large number of customers since it uses loosely linked nodes. Here are a few examples of how the pub/sub architecture may be used:

Internet of Things

The Internet of Things, or IoT, is made up of physical objects that are embedded with technologies like sensors to collect data in real-time. The captured data is then transmitted to other devices and systems via the Internet or other communication networks.

Two main features of pub/sub that make it appealing for IoT use cases include support for flexible coupling between publishers and subscribers, as well as intrinsic support for point-to-multipoint transmission. MQTT and DDS are two well-known protocols that are widely utilized in IoT applications.

Event notifications

A publish-and-subscribe facility is at the heart of an event notification architecture. Sending events to a large number of recipients at the same time is a common technical challenge with the traditional request-response pattern.

For example, a client or message recipient may not always be available, so you need a system that allows them to receive the messages they may have missed. This doesn’t just exist in the request-response pattern. Thanks to the pub/sub architecture’s loose coupling, publishers can send events without worrying about which clients are online because these messages are stored in the subscribing queue.

Data streaming

Data streaming or streams is the technique of transmitting a continuous stream of data. A data stream is a sequence of data pieces that are arranged in time. These signals are constantly flowing into the stream. As a result, data streaming may be described as both continuous and time-sensitive. Pub/sub is ideal for data streaming because it allows computers to interact and react to data updates as they happen.

Pub/sub messaging brokers

A message broker is a computer program module that enables message validation, communication, and routing between applications, systems, and services. Message brokers serve as middlemen between nodes, thus facilitating the exchange of messages between publishers and subscribers.

The primary purpose of a broker is to take incoming messages from applications and perform some action on them. Message brokers effectively implement decoupling by minimizing the mutual awareness between nodes.

The following are some functions of pub/sub messaging brokers:

• Manage workload queue
• Provide reliable storage
• Guarantee message delivery
• Route messages to their various destinations

The broker is a single point of failure, which is a significant downside of employing a message broker. If the message broker fails, the entire system ceases to function.

The pub/sub pattern is a general guideline, so it doesn’t specify the message broker’s precise implementation details. There are a variety of third-party message brokers. Each of them is a fantastic and effective instrument. Let’s review a few of these in no particular order.

Apache Kafka

Apache Kafka is an open-source distributed event streaming platform that aims to provide a unified, high-throughput, low-latency platform for handling real-time data. Companies like Uber, Spotify, Shopify, and Slack use Kafka because it lends itself well for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Active MQ

Apache ActiveMQ is one of the most popular open source, multi-protocol, Java-based message brokers, according to its official documentation. It is flexible enough to accommodate any messaging use-case. Also, it supports MQTT, which you can use to manage your IoT devices. Active MQ also provides advanced features like message load-balancing, mirrored queues, and AMQ message store, which guarantees faster persistence.

Redis

According to its documentation, Redis is an open-source, in-memory data structure store, used as a database, cache, and message broker. Redis supports a wide variety of data structures like strings, lists, maps, sets, sorted sets, HyperLogs, bitmaps, streams, and spatial indices.

ZeroMQ, ØMQ, or 0MQ

ZeroMQ is an asynchronous messaging library that is used to implement messaging and communication systems between applications and processes. These applications can be either in a distributed or concurrent fashion. ZeroMQ provides a message queue that can run with or without a dedicated message broker. It is no wonder that companies like Microsoft, Samsung, AT&T, Auth0, and Bitcoin use ZeroMQ.

Conclusion

In this article, we reviewed the pub/sub pattern, an asynchronous messaging architecture in which messages are exchanged between nodes without knowing the identity of either the sender or the receiver. Some nodes publish messages and are therefore called publishers.

Other nodes subscribe to receive certain messages based on interest and are therefore called subscribers. The event bus/broker refers to nodes that facilitate the exchange of messages between publishers and subscribers.

Keep in mind that the pub/sub architecture is not a one-size-fits-all solution. Request/response models are still relevant and may be utilized in a number of situations.

However, the pub/sub architecture is well-suited for situations where system agility and interoperability are critical. For example, you must continually monitor the resource state and get information about any changes and updates with the shortest possible time latency.

There are numerous additional options for implementing the pub/sub architecture, like Rabbit MQ, IBM MQ, and Faye. I recommend investigating these and selecting the one that best fits your requirements. I hope you enjoyed this article. Feel free to leave a comment if you have any questions.