One of the most fundamental techniques for computers in a network to interact with one another is request/response or request/reply. It is widely utilized by HTTP, which is the World Wide Web’s basis for data transfer. HTTP follows the traditional client-server architecture, with a client initiating a connection to make a request, then waiting for a response.
While this client-server architecture is incredibly helpful and has many applications, it does have certain limits. For example, when communication is asynchronous or includes numerous nodes, the request/reply pattern becomes more difficult.
As a result, developers need an architecture pattern that can function asynchronously and scale well independently of the number of nodes. In this tutorial, we’ll explore the pub/sub architectural pattern, as well as the best message brokers for implementing it. Let’s get started!
Pub/sub, an abbreviation for publish/subscribe, is an asynchronous messaging architecture in which messages are transferred between entities without the sender or the recipient knowing the identity of the other.
In other words, the pub/sub architecture’s two sides are loosely connected, making it an appropriate design for integrating the nodes of an expanding distributed system. Pub/sub allows machines to interact and respond to data updates in real-time, which differs from the usual request/response communications paradigm in which data is updated at regular intervals.
There are three main components that make up the pub/sub architecture, publishers, the event bus or broker, and the subscribers. Let’s define these.
Publishers are nodes that generate messages that are subsequently sent across the system using the event bus/broker.
The event bus/broker nodes serve as middlemen, facilitating the flow of messages from publishers to subscribers. Brokers further strengthen the decoupling between system nodes because subscribers deal with the broker rather than the entire system.
Subscribers essentially listen for communication about the topics and categories in which they are interested, doing so without knowing who the senders of these communications are.
Subscribers typically indicate their interest in receiving specific messages, acting as a filtering mechanism. Because of the fine-grained topic control, it’s simple to ensure that the various event buses are sending the correct message. We can achieve this type of filtering using either topic-based filtering or content-based filtering.
Topic based filtering requires the messages to be disseminated into logical channels. The subscribers only get messages from logic channels to which they have subscribed.
Content-based filtering enables subscribers to receive messages based on the substance of the messages. The subscribers have filters that check whether the content of the messages being broadcast matches the constraints defined by the subscriber.
The pub/sub architecture is a great option for distributed architectures with a large number of customers since it uses loosely linked nodes. Here are a few examples of how the pub/sub architecture may be used:
The Internet of Things, or IoT, is made up of physical objects that are embedded with technologies like sensors to collect data in real-time. The captured data is then transmitted to other devices and systems via the Internet or other communication networks.
Two main features of pub/sub that make it appealing for IoT use cases include support for flexible coupling between publishers and subscribers, as well as intrinsic support for point-to-multipoint transmission. MQTT and DDS are two well-known protocols that are widely utilized in IoT applications.
A publish-and-subscribe facility is at the heart of an event notification architecture. Sending events to a large number of recipients at the same time is a common technical challenge with the traditional request-response pattern.
For example, a client or message recipient may not always be available, so you need a system that allows them to receive the messages they may have missed. This doesn’t just exist in the request-response pattern. Thanks to the pub/sub architecture’s loose coupling, publishers can send events without worrying about which clients are online because these messages are stored in the subscribing queue.
Data streaming or streams is the technique of transmitting a continuous stream of data. A data stream is a sequence of data pieces that are arranged in time. These signals are constantly flowing into the stream. As a result, data streaming may be described as both continuous and time-sensitive. Pub/sub is ideal for data streaming because it allows computers to interact and react to data updates as they happen.
A message broker is a computer program module that enables message validation, communication, and routing between applications, systems, and services. Message brokers serve as middlemen between nodes, thus facilitating the exchange of messages between publishers and subscribers.
The primary purpose of a broker is to take incoming messages from applications and perform some action on them. Message brokers effectively implement decoupling by minimizing the mutual awareness between nodes.
The following are some functions of pub/sub messaging brokers:
The broker is a single point of failure, which is a significant downside of employing a message broker. If the message broker fails, the entire system ceases to function.
The pub/sub pattern is a general guideline, so it doesn’t specify the message broker’s precise implementation details. There are a variety of third-party message brokers. Each of them is a fantastic and effective instrument. Let’s review a few of these in no particular order.
Apache Kafka is an open-source distributed event streaming platform that aims to provide a unified, high-throughput, low-latency platform for handling real-time data. Companies like Uber, Spotify, Shopify, and Slack use Kafka because it lends itself well for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Apache ActiveMQ is one of the most popular open source, multi-protocol, Java-based message brokers, according to its official documentation. It is flexible enough to accommodate any messaging use-case. Also, it supports MQTT, which you can use to manage your IoT devices. Active MQ also provides advanced features like message load-balancing, mirrored queues, and AMQ message store, which guarantees faster persistence.
According to its documentation, Redis is an open-source, in-memory data structure store, used as a database, cache, and message broker. Redis supports a wide variety of data structures like strings, lists, maps, sets, sorted sets, HyperLogs, bitmaps, streams, and spatial indices.
ZeroMQ is an asynchronous messaging library that is used to implement messaging and communication systems between applications and processes. These applications can be either in a distributed or concurrent fashion. ZeroMQ provides a message queue that can run with or without a dedicated message broker. It is no wonder that companies like Microsoft, Samsung, AT&T, Auth0, and Bitcoin use ZeroMQ.
In this article, we reviewed the pub/sub pattern, an asynchronous messaging architecture in which messages are exchanged between nodes without knowing the identity of either the sender or the receiver. Some nodes publish messages and are therefore called publishers.
Other nodes subscribe to receive certain messages based on interest and are therefore called subscribers. The event bus/broker refers to nodes that facilitate the exchange of messages between publishers and subscribers.
Keep in mind that the pub/sub architecture is not a one-size-fits-all solution. Request/response models are still relevant and may be utilized in a number of situations.
However, the pub/sub architecture is well-suited for situations where system agility and interoperability are critical. For example, you must continually monitor the resource state and get information about any changes and updates with the shortest possible time latency.
There are numerous additional options for implementing the pub/sub architecture, like Rabbit MQ, IBM MQ, and Faye. I recommend investigating these and selecting the one that best fits your requirements. I hope you enjoyed this article. Feel free to leave a comment if you have any questions.
Monitor failed and slow network requests in productionDeploying a Node-based web app or website is the easy part. Making sure your Node instance continues to serve resources to your app is where things get tougher. If you’re interested in ensuring requests to the backend or third-party services are successful, try LogRocket.
LogRocket is like a DVR for web and mobile apps, recording literally everything that happens while a user interacts with your app. Instead of guessing why problems happen, you can aggregate and report on problematic network requests to quickly understand the root cause.
LogRocket instruments your app to record baseline performance timings such as page load time, time to first byte, slow network requests, and also logs Redux, NgRx, and Vuex actions/state. Start monitoring for free.
Install LogRocket via npm or script tag. LogRocket.init() must be called client-side, not
server-side
Learn more →
$ npm i --save logrocket
// Code:
import LogRocket from 'logrocket';
LogRocket.init('app/id');
// Add to your HTML:
<script src="https://cdn.lr-ingest.com/LogRocket.min.js"></script>
<script>window.LogRocket && window.LogRocket.init('app/id');</script>
Vite is a versatile, fast, lightweight build tool with an exceptional DX. Let’s explore when and why you should adopt Vite in your projects.
Explore advanced capabilities for content sharing with the navigator.share API, including dynamic content sharing, custom share targets, and batch sharing.
We spoke with Chas to get his insights on building technology and internal processes for companies that are scaling quickly.
Cypress is one of today’s foremost tools for testing web applications. Let’s explore when and why you should adopt Cypress in your projects.
2 Replies to "Best pub/sub messaging brokers"
One thing that always confuses me is how does a web app subscribe to a certain topic or listen for an event notification that is only relevant to the logged in user?
If the web app is receiving every message but only has an ‘IF user === loggedInUser’, wouldn’t someone be able to see every other unfiltered message by opening DevTools?
Late answer, but the answer is simple: Web clients does not receive every message over wire. This is filtered server side. The web clients rarely connect directly to the broker, but receives the data over a web socket. For machine to machine clients, that do connect directly to a broker, these things can be controlled via subscriptions.