In theory, sending a message over a wire to another device is a simple task. But sending a message in a reliable and secure way can be a pretty hard job. In an IoT era, where every day the number of devices that are connected to the internet increases drastically, we need to find different communication mechanisms.
Because we cannot control when a device is connected to the internet and ready to receive our package, we need to find different ways to communicate with it.
In this post we will look at different messaging systems that are offered by Microsoft Azure. For each messaging system we will try to identify its strengths and when we should use it. Once we understood each messaging system, we will compare them one by one.
In the end of this post we will try to find a perfect messaging system that can be used in any situation. For different use cases we may need to use different messaging systems, based on our needs.
The messaging solutions that will be discussed in this post are:
This messaging system is part of Azure Storage and allows us to store a lot of messages in the same queue. When I say a lot, imagine queues that can reach not 1 GB, not 1TB and not 10TB. Using Azure Storage Queues we can have a queue that has 200 TB. Because of this, we have the ability to store large amounts of data in queues, without thinking about the size of the queue.
Another benefit of this type of queue is the number of concurrent clients that in theory is unlimited. The only limit in this case is the bandwidth that can limit the number of concurrent clients. The maximum size of a message is 64 KB, but in combination blobs we can have messages that reach 200 GB.
Of course there are also some limitations that we need to take into account. First of all, even if the size of queue can be very large, the maximum TTL (Time To Leave) of a message is 7 days. This means that a message needs to be consumed in 7 days or renewed; otherwise the message will be removed.
Even if we have support for base messaging capabilities like scheduler delivery and batch support, we don’t have support for state management, duplicate detection or transaction support.
An interesting feature of Azure Storage Queues is the logging capabilities. Users have the ability to activate the loggings mechanism and track all the actions that are happening on the queue. Tracking information like client IP are tracked and stored as an out of the box solution.
Clients have the ability to only peek at the messages from the queue, without removing or locking them. This means that if a client peeks at a message, other clients will be able to access the same message from the queue.
Because of this, Azure Storage Queues are great when you need a messaging system that is able to track all the actions that are happening on it. It can be a good solution for use cases when you know that the size of the queue will be bigger than 80-100 GB. For large queues, this can be the best queue mechanism.
This messaging system is part of a more complex messaging infrastructure offered by Microsoft and supports more complex scenarios. Because of this, we will discover that the size of a queue is limited to 80 GB, but the features offered by Service Bus Queues are more complex.
It is important to know from the start that these two messaging systems are constructed over different services and have nothing in common. The size of a message can be 256 KB, larger in comparison with Azure Storage Queues and also a message will be persisted in Service Bus for an unlimited period of time. On top of this, there is full support for AMPQ protocol, feature that can be very useful for embedded devices.
There is support for dead lettering, which allows us to move automatically a message in a secondary queue, if the message expires or clients cannot consume a specific message. There is full support for transactions, handling a specific number of messages in the same transaction. On top of this, there is support to group multiple messages in the same session – in this way, we can ensure that a client will consume all messages that are part of a specific session. If we know the ID of the session, we can consume messages from that specific session.
An interesting feature of Azure Service Bus Queues is duplicate detection. Once it is activated, we can detect duplicate messages. The moment we want to add a message that already exists in the system, the message will not be added. This is great when we want to ensure that we have unique messages in the queue.
Messages can be consumed from the queue in two different ways – Peek and Lock or Receive and Delete. We can peek at a message from a queue and make it unavailable for the rest of the clients until we confirm that we consumed it with success or abort the action (we can specify also a timeout).
The security capabilities of Azure Service Bus Queues are more complex, we have the ability to control the access to the messages more deeply.
Based on these features, Azure Service Bus Queues are great when we need duplicate detection, transaction support or to store messages for an unlimited period of time.
In contrast with Azure Service Bus Queues that allow us to deliver one-to-one, Azure Service Bus Topics allow us to deliver messages one-to-many. This means that we can deliver the same message to multiple clients that are called subscribers. This messaging system is basically an ESB system (Enterprise Service Bus), allowing to have a “publish/subscribe” communication.
From the perspective of the features, they are very similar with Azure Service Bus Queues with some additional features and capabilities. The similarity of these two services is because both messaging systems are constructed over the same broker messaging infrastructure system.
Each topic that is used to send messages can have maximum 2.000 subscribers. This means that the same message can be received by 2.000 subscribers. This can be very useful when we need to distribute messages to different systems. Also, a subscription can be added at runtime. We don’t need to stop the system or recreate the topic. Once a new subscription is created, all the new messages that are sent to that topic will be received by the new subscription also.
An interesting feature is the filter support. We can attach a filter to each subscription. That filter will allow only the messages that respect the filter rule to reach that specific subscription. In this way, each subscription can listen to specific messages.
Both messaging systems, Azure Service Bus Topics and Queues have automatic forward capabilities, but for Topics, they are more interesting. This means that we can forward a message from a subscription automatically to another Service Bus Topic of Queue.
Each message that is sent over a Service Bus can have a collection of properties. Properties are used when a custom filter per subscription is applied. Also, each subscription can have a custom action that is executed when a message is received by it. Even if the actions that can be executed are very simple, it can be very useful in some situations – for example, we are allowed to change the name of the value of the property.
Based on these capabilities, Azure Service Bus Topics are perfect to use when we need to distribute a message to multiple listeners. In systems where the number of clients that should receive a message can change dynamically, Azure Service Bus Topics can be useful.
Even if, in theory, it is part of Azure Service Bus, Azure Event Hub is a special messaging system that is used to ingest large amounts of data in a short period of time. This system is capable to ingest more than 1 million messages per second without any kind of problem.
It is constructed around a streaming concept. Because of this, all messages that flow in the system are seen like a stream of data. The latency is very low and even if the quantity of data that flows is very high the system is reliable and stable.
An interesting feature of this system is the capacity to navigate between the messages that we already received. We have a concept similar with a cursor and we can iterate over old messages also – reply capability. On top of this, the stream of messages can reach multiple consumers in the same time using the consumer group concept.
An important feature that can be found on Azure Service Bus is the message order preserving. It means that the order of messages is persisted. Consumers will be able to consume messages in the order they were sent.
The Event Hub capabilities can scale pretty interesting, by adding multiple TUs (Throughput Units). Each TU allows us 1 MB/s for ingress and 2 MB/s for egress. The retention of a message is 1 day, being able to store them up to 7 days.
Of course, this comes with some costs. Features like death letter queue, transaction support or TTL options cannot be found on Azure Event Hub.
This solution is perfect for high volumes of messaging processing, like telemetry or in IoT use cases.
There is no unique response for this question. Based on our needs and requirements, we may use different messaging systems. Azure Storage Queues are perfect when we need to store and manage large amounts of messages but, when we need more control, Azure Service Bus Queues can be a better solution. For use cases when we need to distribute messages to multiple listeners, Azure Service Bus Topic is the best. But, for large amounts of messaging processing, nothing can beat Azure Event Hub.
In this post, we looked at different messaging systems offered by Azure and identified the most important features of each of them. We saw the most important use cases when we can use these systems and what the weak points are. At the end, I would like to invite you to go to Azure web site and discover more about these great services.