1. Message Queuing

Message queuing is a mechanism by which sender sends a message to receiver asynchronously. Sender and receiver may or may not interact with the message at the same time. Then message is sent to a queue where it is stored until the receiver retrieves the message. These queue(s) can be on single or multiple servers and can reside in-memory or persisted on disk(durable).

AMQP is feature-rich open standard message queuing protocol for messaging and was developed by JPMorgan Chase and IMatix. Some of the AMQP Message Brokers are RabbitMQ and Windows Azure Service Bus.

RabbitMQ is an implementation of the AMQP message broker and queuing System. It is an open source and allows to integrate systems together that are robust, reliable and scalable. RabbitMQ server, which is built upon the Erlang programming technologies, is a cross-platform system which can run on Windows Servers, Linux, Unix systems (Ubuntu, Solaris), Mac OS X and cloud platforms like Amazon EC2,Microsoft Azure. Rabbit MQ also has client libraries in different programming languages such as Microsoft .NET, Java, Erlang, Ruby, Python, PHP, Perl, C/C++, Node.JS

As per above figure

1) Sender application with AMQP protocol sends message to exchange inside message broker

2) Message broker is like a mailbox

3) Exchange then sends message to queue based on rules

4) The message is delivered from the queue to a receiver application

5) The receiver application pulls the message that is subscribed to the queue

When sender sends a message to message broker, the messages are sent to exchanges which are AMQP entities. Exchanges take a message and then route it to one or more queues. The type of routing depends upon the exchange type used and different exchange rules called bindings. These are four type of exchanges

• Direct
• Fanout
• Topic
• Headers

Each exchange is also declared with a number of different attributes:

Name : Name of the exchange

Durability: Determines whether or not messages sent to the exchange survive a broker/server restart by persisting the messages to disk. Options are Durable/Transient Auto-delete:Determines if the exchange is deleted when all of the queues have finished using it.Options are Yes/No.

Internal :If yes, clients cannot publish to this exchange directly. It can only be used with exchange to exchange bindings.Options are Yes/No.

Arguments: These are message broker-dependent.

In this the exchange delivers message to queue based on routing key.A direct exchange works as follows :

1) A queue binds to the exchange with routing key for example "Salaryprocessing_message"

2) Sender application sends the message to exchange with routing key = "Salaryprocessing_message"

3) Exchange matches the routing key send by sender application and it routes the message to appropriate queue

4) Receiving application which are subscribed to the specific routing key receives the message. Direct queues are commonly used to distribute messages between multiple worker processes in a round robin manner.

In fanout exchange, routing key is ignored. Once message sent to fanout exchange, it routes all the messages to queues that are bound to it. So if a message needs to be broadcast to all of the queue consumers then fanout exchange is can be used.

Topic exchanges route messages to one or many queues based upon matching between a message routing key and the pattern that was used to bind a queue to an exchange.Topic exchanges are commonly used for the multicast routing of messages to different queues.

Steps:

A message is sent to an exchange from the client along with a routing key.

i) The message will be sent to a named exchange of type topic.

ii) A copy of the message will be sent to queues where they have a binding which matches an expression against the routing key.

iii) Each queue will then have a set of receivers, which will be able to receive messages from that queue. Topic exchanges have a broad set of use cases. Whenever a problem involves multiple consumers/applications that selectively choose which type of messages they want to receive, the use of topic exchanges should be considered.

A topic exchange works by using a wildcarded routing key. The routing key will be a piece of information, which is used to direct messages, and the routing key in the subscription can be replaced with special characters.

a) * can replace one word

b) # can replace zero or more words

Let's take a look at this pattern in action. In this case the sender will send a message to an exchange, which is of type topic. The message will include a routing key containing words separated by a dot. In this case, the sender will publish a message with a routing key which contains the Part1, then a dot, then the Part2, then a dot, then the Part3.

From the exchange there will be four queues which have specified a binding to that exchange.

i) The first queue is interested in all Part1 and Part2. This means that it has used word Part1, then a dot, then the word Part2, followed by a dot, then the word star.

ii) The second queue is interested in Part1 ,Part3 and anything.

iii) The third queue is interested in Part1 and anything.

iv) The message from exchanged will not be routed to fourth queue as it is interested in Part1,Part2, Part4 but Part4 is not in message sent by sender.

This pattern is very similar to the topic pattern, except that instead of evaluating the routing key, it will be looking at headers on the message and then determining who is interested in those messages.

This means that the routing key is ignored for this type of exchange due to the fact that it can only express one piece of information. You can assign multiple header values that pass into the exchange and are routed to the queues.

A message will be sent from the sender to an exchange, and the message would include some headers. Each queue would be bound to that exchange with a binding that would expect to match against some of the headers. If the exchange identifies that a binding matches the message headers, a copy of that message will be placed on the queue. Each queue would then be processed by a number of receivers.

The exchange would be of type headers. The routing key for the message would not be set, but instead the message would have some properties specified in the message headers. 

Let's take an example above.

x-match= any

Here header is set to x-match= any, the message will be routed to first and second queue because atleast one header is matching . Message will not be routed to third queue as none of the headers are matching.

x-match= all

Here header is set to x-match= all , the message will be routed to first queue because both property of header is matching . Message will not be routed to second and third queue as both properties of header are not matching.

In this pattern, the message sender will create a temporary response queue and start polling it. It will then send a message to an exchange. The message will be copied to queues, which have a matching binding, and on those queues receivers will take the message and process it. Each receiver will send a response to the temporary response queue as specified in the message headers. The sender will then get a collection of responses from that temporary response queue.

We start with an exchange, which has a binding to two queues in RabbitMQ. Here we have two receivers that are listening on those queues. The sender would be about to send a message. Before the sender sends a message, it would create a temporary response queue, and then it would send the message to RabbitMQ. The exchange would forward the message to the appropriate queues, depending on the routing rules that have been implemented and then both the receivers get that message from the appropriate queue. Both the receivers will send a response message to the temporary response queue, which will then go back to the original message sender.

Queues in AMQP are similar to queues in any other messaging system. Messages are placed onto a queue and they work on a first in, first out (FIFO) basis. Before you can use a queue, it must be declared. If the queue doesn’t already exist, it will be created. If the queue already exists, then re-declaring the queue will have no additional effect on the queue that already exists. Queue names can be picked by the application or they can be automatically named by the broker that generates them.

Queue names can be up to 255 characters in length. If you want the broker to pick the queue name for you, then you don’t pass a name at the point where you declare the queue.

Queues can be made durable which means the queue is persisted to the disk. This only makes the queue persistent and not the messages. Queue durability means the queue will be re declared once the broker is restarted. If you want the messages to be also persisted, then you have to post persistent messages. This is useful if you need the messages to remain intact if the broker or server is restarted. Making queues durable does come with additional overhead so you need to decide if you need this enabling but, if your application can’t be in a position where it can lose messages, then you need queue durability.

Queues have the following additional properties over exchanges:

Name: The name of the queue.

Durable: The queue and messages will survive a broker or server restart. options are Durable/Transient.

Auto Delete: The queue is deleted when the consumer or subscriber unsubscribes. Message TTL : How long a message published to a queue can live before it is discarded (milliseconds).

Auto Expire : How long a queue can be unused for before it is automatically deleted (milliseconds).

Max length :How many (ready) messages a queue can contain before it starts to drop them from its head.

Dead letter exchange : Optional name of an exchange to which messages will be republished if they are rejected or expire.

Dead letter routing key :Optional replacement routing key to use when a message is dead-lettered. If this is not set, the message's original routing key will be used.

When you want to define rules that specify how messages are routed from exchanges to queues, you need to define bindings. Bindings may have an optional routing key attribute that is used by some exchange types to route messages from the exchange to the queue. The purpose of the routing key is to select certain messages published to an exchange to be routed to the bound queue. This means that the routing key acts like a filter.

If an AMQP message cannot be routed to any queue because it does not have a valid binding from the exchange to the queue, then it is either dropped or returned to the publisher depending upon message attributes the publisher has set.

When a consuming application acts on a message from the queue, it is possible that a problem could occur in that application which means that message is lost. Generally, when an application acts on a message, that message is removed from the queue but you may not want this to happen until you have successfully processed the message. The AMQP protocol gives you a couple of ways of defining when a message is removed from the queue.

Automatic acknowledgement :The message is removed once the broker has sent the message to the application.

Explicit acknowledgement: The message is removed once the application has sent an acknowledgement message back to the broker. Here it is up to the application to decide when to remove the message from the queue. This could be when you have just received the message or after you have processed it.If the consuming application crashes before the acknowledgement has been sent, then the message broker will try to redeliver the message to another consumer.

When an application processes a message, that processing may or may not succeed. If the processing fails for any reason (for example, there is a database time-out), then the consuming application can reject the message. When this happens, the application can ask the broker to discard the message or re-queue it.

Durability: Messages can either be kept in memory or persisted to disk. If they are persisted to disk, then the message will be preserved should the server containing the queues crash. If this happens, once the server comes back up, the receiving applications can pick up where they left off.

Security: A message broker should be able to let you set up different security permissions for different users. Some applications may have permission to write but not read from the queue, or another application could just have permissions to read from the queue.

Time to Live: A message may have a time to live property set on it. This will mean a message can be set to expire after a certain time span.

Filtering and Routing: A messaging system might have the ability to filter the messages to decide to which queue a message should be delivered.

Batching: Batching allows messages to be saved up and then delivered all at the same time as part of a larger batch; this is typically used when a destination server is not always online.

Receipt Acknowledgement: A message publisher may require an acknowledgement from the message broker that the message has been successfully delivered

Message queuing is an important architectural technique for creating scalable and decoupled solutions.