Routing mechanisms for AMQP protocol

In the previous article, we installed the Qpid Dispatch Router and had a quick overview about the available tools inside the installation package for both router management and monitoring.

Now it’s time to start using the router from simple to complex configurations and network topologies with some examples which will involve AMQP senders, receivers and/or brokers. The broker part won’t be always necessary because the AMQP protocol is a “peer to peer” protocol and it works great connecting two clients directly without the “store and forward” mechanism provided by a broker in the middle. For more information about that you can read the first article of this series.

In this article, I’ll use the router with the default configuration showing how a sender and a receiver can connect to it and exchange messages through the router itself.

Routing mechanisms

First of all it’s interesting to say that the router supports two different types of routing mechanisms.

Message routing

When the router receives a message on a link, it uses the address specified in the target terminus when the sender attached the link itself to the route; if this address wasn’t specified, the destination address is get from the “To” property of the message. Based on such information, the router inspects its routing table to determine the route for delivering the message : it could be a link attached by a direct receiver to the router or another router inside the network that will be the next hop for reaching the destination. Of course, the message could be sent to different receivers all interested in the same address. The main point here is that the routing decision is made for each received message and there is always a communication between internal router nodes and external clients.

As you can see in the above picture, a link is established between sender and router and between router and receiver. They are two completely distinct links that the router uses for messages exchange between sender and receiver through the routing mechanism on message basis.

For example, it means that there is a different flow control between router (with its internal receiver) and sender and between router (with its internal sender) and receiver : of course, it’s true that if the receiver grants few credits (i.e. 10) but the router (the internal receiver) grants more credits to the sender (i.e. 250 by default), it takes in account this difference. If for any reason the receiver closes the connection (after receiving 10 messages) and sender has already sent more than 10 messages (acknowledged by an “accepted” disposition), the router will reply with a “released” disposition for the next 40 messages because they can’t be delivered to the closed receiver.

Another interesting point is related to the message “settlement” : the router always propagates the delivery and its settlement along the network. On receiving a “pre-settled” message, its nature is propagated to the destination. The same is for “unsettled” messages : in that case, the router needs to track the incoming delivery and send the unsettled message to the destination; when it will receive the disposition (settlement) from the final receiver, it will reply in the same way to the original sender.

The last interesting aspect of message routing is the available routing patterns which define the paths followed by messages across the network :

• closest : even if there are more receivers for the same address, the message is sent on the shortest path to reach the destination. It means that only one receiver will get the message;
• balanced : when more receivers are attached to the same address, the messages sent to that address are spread across receivers. It means that each receiver will receive a different message at time in a sort of “competing consumers” way;
• multicast : it’s something like a “publish/subscribe” pattern where all the receivers will receive the same message published on the address they are attached;

When the receivers for a specific address are all connected to the same router, we could think that “closest” and “balanced” have the same behavior because all the paths have same length and receivers are closed at same level to the router. It’s not so true because with “closest” the router uses a strict round-robin distribution across receivers while with “balanced” it takes into account the number of unsettled deliveries for each receiver and favors the “faster” of them (who settled messages faster than others).

To be more clear, suppose to have a sender S, two receivers R1 and R2 and a routers network with R1 connected to the same router as S and R2 connected to a different router (connected to the previous). We can say that R1 is “one level closed” to S and R2 is “two level closed” to S.

In the following scenario, with the “closest” distribution all the messages sent by S will be always delivered to R1.

“closest” message routing

Using the “balanced” distribution, the messages are spread across both receivers and there is nor relation with the path length.

“balanced” message routing

Finally, with “multicast” distribution all messages are sent to all receivers.

“multicast” message routing

Link routing

When the router receives an attach request, it’s propagated through the network in order to reach the destination node and establish the real link. When the sender starts to send messages to the router, it propagates that message through the established link to the destination without making any decision at message level. You can think of it as a sort of virtual connection or a tunnel between sender and receiver through a routers network.

As you can see in the above picture, the link is established directly between the two peers and all performatives go through it.

From a flow control point of view, it’s directly handled between sender and receiver; any credits granted by the receiver arrives to the sender with a flow performative without any interference by the router in the middle. The link through the router is like a “tunnel” and it seems that the two peers are directly connected.

The same is true for disposition about settlement for “unsettled” messages that the sender receives directly from the receiver.

The concept of different routing patterns doesn’t make sense because in this case there is a direct link between sender and receiver so the router doesn’t make any decision on single message basis but it has only to propagate the frame along the link.

Let’s start … simple !

Now it’s time for a very simple example with a router started using the default configuration and only one sender and one receiver connected to it and attached to the “/my_address” address.

I used the “simple_recv” and “simple_send” C++ client examples for that and you can find them inside the Qpid Proton installation folder.

First of all let’s start the receiver specifying the address and the number of messages it wants to receive (it will grant link credits for that), i.e. 10 messages.

Using the qdstat management tool we can see that an endpoint for the “my_address” address is defined with “out” direction (from router to the receiver) with no messages delivered yet.

After that let’s start the sender in order to send some auto generated messages, i.e. 5 messages.

As you can see, the messages sent are all settled and confirmed. What’s happened at receiver side ?

All messages sent by sender are now received and the simple application doesn’t close the connection because it’s waiting for the other 5 messages in relation to the 10 credits granted (of course it’s only an application behavior and not related to the router mechanisms).

Inspecting the router with the qdstat management tool we can see that on the output endpoint for the “my_address” address there are 5 delivered messages. What we can’t see is the endpoint on the same address with the opposite “in” direction (from sender to router) because after sending the messages, the sender closed the connection to the router and the endpoint is deleted. You can trust me … it was alive for all the time the sender was sending messages !

As you can see we have directly connected sender and receiver without the need for a broker in the middle with its “store and forward” mechanism. In the above scenario, when the messages are settled and confirmed to the sender it means that they are really received by the receiver.

Conclusion

With this article I introduced the different mechanisms for messages routing that the Qpid Dispatch Router provides. For every scenarios we can choose the better way and what we need in order to distribute messages in a useful way for our distributed application. We saw a simple example on connecting sender and receiver through the router without the need for a broker in the middle. In the next articles, I’ll increase the complexity starting to use non default configuration files and exploring different way to connect routers with clients and brokers.