AMQP

Vert.x and IoT in Rome : what a meetup !

Yesterday I had a great day in Rome for a meetup hosted by Meet{cast} (powered by dotnetpodcast community) and Codemotion, speaking about Vert.x and how we can use it for developing “end to end” Internet of Things solutions.

17352445_10208955590111131_6229030843024604532_n

17352567_10208955588791098_766816304298598626_n

I started with an high level introduction on Vert.x and how it works, its internals and its main usage then I moved to dig into some specific components useful for developing IoT applications like the MQTT server, AMQP Proton and Kafka client.

17342690_10208955588751097_8818320599257580571_n

17352571_10208955588951102_2851165399929439718_n

It was interesting to know that even in Italy a lot of developers and companies are moving to use Vert.x for developing microservices based solutions. A lot of interesting questions came out … people seem to like it !

Finally, in order to prove the Vert.x usage in enterprise applications I showed two real use cases that today work thanks to the above components : Eclipse Hono and EnMasse. I had few time to explain better how EnMasse works in details, the Qpid Dispatch Router component in particular and for this reason I hope to have a future meetup on that, the AMQP router concept is quite new today ! In any case, knowing that such a scalable platform is based (even) on Vert.x was a great news for the attendees.

17264802_10208955590191133_8923182437405273553_n

If you are interested to know more about that, you can take a look to the slides and the demo. Following the link to the video of the meetup but only in Italian (my apologies for my English friends :-)). Hope you’ll enjoy the content !

Of course, I had some networking with attendees after the meetup and … with some beer 🙂

17310150_1421561734583219_8414988688301135801_o

“Reactive Internet of Things : the Vert.x way” … meetup in Rome !

vertx_iot

On March 16th I’ll be guest of the Meet{cast} and Codemotion community for a meetup in Rome speaking about … “Reactive Internet of Things : the Vert.x way”.

meetcastcodemotion

It’s a pleasure for me showing how the Vert.x toolkit can be used for developing Internet of Things solutions leveraging on the pillars of the reactive manifesto (responsive, elastic, resilient and asynchronous).

Starting from an introduction on what Vert.x is, what it provides and its main features, I’ll move to the messaging and IoT focused components that the toolkit offers. So we’ll see the new MQTT server and Kafka client (officially in the latest 3.4.0 Beta 1 release) and the well know AMQP Proton and Bridge components. Of course …. demos around them !

Finally, I’ll show how these components are already used today for enterprise IoT solutions introducing the Eclipse Hono project, for handling IoT connectivity, and the EnMasse which provides a Message as a Service platform. The great thing is that we’ll have the chance to see the code because … they are open source of course !

So … what are you waiting for … register for the meetup here ! See you in Rome 😉

Eclipse Hono : “Connect. Command. Control” … even on OpenShift !

The Eclipse Foundation is the main open source community which has a great focus on the Internet of Things world and the related Eclipse IoT ecosystem involves a lot of different projects and partners like Red Hat, Bosch, Eurotech, IBM and many more. Recently, publishing this white paper, they showed a full stack with all the available tools for building a complete IoT solution from devices to the Cloud through the gateways.

selection_005

In relation to the Cloud side, one of the main problems in the IoT world is the ability to handle millions of connections from devices (and gateways) in the field, their registration for managing authentication and authorisation and, last but not least, the ingestion of the data received from them like telemetry or events. Finally, the last point is related to control these devices sending them commands in order to execute actions in the environment around them or upgrading their software and configuration.

The Eclipse Hono™ project is the answer to these problem !

The APIs

From the official web site, we can read :

Eclipse Hono™ provides remote service interfaces for connecting large numbers of IoT devices to a back end and interacting with them in a uniform way regardless of the device communication protocol.

The mantra from the landing page of the web site project is “Connect. Command. Control” which is made a reality through a well defined API for :

  • Registration : for handling the requests related to the registration (so creation) of a new device so that it can be authorised to connect. It’s also possible retrieving information about registered devices or delete them;
  • Telemetry : for the ingestion of a large volume of data from devices (i.e sensors) available for analysis to the backend applications;
  • Event : for receiving specific events (i.e. alarms, notification, …) from devices for making decision on the Cloud side. This API is quite similar to a telemetry path but it uses a “different” channel in order to avoid such events going through the overwhelmed telemetry one;
  • Command & Control : for sending commands to the devices (from a backend application) for executing operations and actions on the field (receiving the related response) and/or upgrading local software and configuration;

All the above APIs are accessible through the AMQP 1.0 protocol (the only standardised AMQP version!) and it means that they are defined in terms of addresses on which devices need to connect for interacting with the system and the properties/content of the messages; of course, it’s true not only for devices but even for business and backend applications which can receive data from devices or send them commands. In any case, it doesn’t mean that devices which aren’t able to speak such protocol can’t connect but they can leverage on the protocol adapters provided by Hono; the current implementation provides an MQTT and HTTP REST adapter.

All these APIs are though in order to allow multi-tenancy so that using a single deployment, it’s possible to handle channels for different tenants so that each of them can’t see data or exchanged messages from the others.

The Architecture

The main components which build the Eclipse Hono™ architecture are :

  1. Protocol Adapters : these components adapt a device protocol to the first citizens protocol used in Hono, the AMQP 1.0. Today, an MQTT and HTT REST adapters are provided out of box but thanks to the available interfaces, the user can develop a new adapter even for some custom protocols;
  2. Hono Server : this is the main component to which devices can connect directly through AMQP 1.0 or through the protocol adapters. It’s in charge to expose the APIs in terms of endpoints and handling the authentication and authorisation of devices;
  3. Qpid Dispatch Router : this is an AMQP 1.0 router, part of the Apache Qpid project, which provides the underlying infrastructure for handling millions of connections from devices in the fields. The simpler deployment can use only one router but in order to guarantee reliability and high volume ingestion, a router network should be used;
  4. ActiveMQ Artemis : this is the broker mainly used for handling command and control API so for storing commands which have to be delivered to devices;

While the devices connect directly to the Hono Server (or through protocol adapters), the backend applications connect to the Qpid Dispatch Router leveraging on direct addressing for receiving telemetry data (if no application is online, no devices are able to send data) or sending commands (the queus backed in the broker are reachable through the router).

selection_004

The running environment

All the artifacts from the project are provided as Docker images for each of the above components that can run using Docker Compose (the Docker Swarm support will be available soon) or using a more focused Cloud platform like OpenShift (compatible with Kubernetes deployment as well).

selection_006

Regarding the OpenShift deployment, the building process of the Hono project provides a bunch of YAML files which describe the objects to deploy like deployment configurations, services, persistent volumes and related claims. All the need is having an OpenShift platform instance accessible and deploy such objects for having Eclipse Hono™ running in a full featured Cloud environment with all the related scaling capabilities.

hono_openshift

The example deployment is based on four pods, one for each of the main components so there is the router pod, the Hono Server pod and one pod for each protocol adapter; of course if you need the command & control path, even the broker pod need to be deployed.

In order to try it, an OpenShift Origin instance can be used on a local PC for deploying the entire Hono solution; for example, the above picture is related to my tests where you can see all the pods running on OpenShift (left side) with simulated devices that interact using MQTT and HTTP REST (on the right side).

The documentation which describes the steps for having such a deployment is available on the official web site here.

So what are you waiting for ? Give it a try !

Conclusion

In my mind every IoT solution should be made of three different layers (a little bit different from the Eclipse vision) : the devices/gateways, the connectivity layer and the service layer.

While the Eclipse Kura project fits in the gateways layer and the Eclipse Kapua in the service layer, Eclipse Hono is the glue between them in order to handle the connections from devices and making available their data to the backend services (and vice versa in the opposite direction for command and control). Thanks to the API standardisation  and the usage of a standard protocol like AMQP 1.0, Hono can be used for connecting any kind of devices with any kind of services; of course leveraging on a these three project has the big advantage of having big companies working on them, mainly Red Hat, Bosch and Eurotech.

Finally, the solution is always the same …. open source and collaboration ! 😉

 

Designing MQTT over AMQP

mqtt_over_amqp

Let’s think about what we can consider some weaknesses of the MQTT protocol …

  • it provides native publish/subscribe pattern only; having request/reply is quite cumbersome with “correlation injection” inside topics and payload messages;
  • it doesn’t provide flow control;
  • it doesn’t have metadata information inside messages (i.e. content type);
  • it’s fully broker centric;

but there are some unique features that we have to consider as strengths for this protocol as well …

  • retain message also known as “last well known” message;
  • last will testament (LWT);
  • session handling;

Now, let’s think about the main features that fill the gap for the MQTT protocol but provided by AMQP …

  • native support for both publish/subscribe and request/reply patterns (correlation for free);
  • flow control at different levels (with max window size and message credits);
  • a full type system and metadata information on each message;
  • peer-to-peer model (the broker is just a peer);

but it lacks of the above three MQTT features !

So how greater could be AMQP protocol having such features on top of it ?

Under the open source EnMasse project, I have been working on having a design (so a kind of “specification”) for having retain message, last will testament and session handling over AMQP. At same time I have been developing an MQTT “gateway” in order to allow remote MQTT clients to connect to an AMQP based “Message as a Service” like EnMasse.

Having such a design means that not only an MQTT client can leverage on receiving retain message after subscribing to a topic, sending its LWT on connection or receiving messages for a topic when it was offline; it means having the above features for native AMQP clients as well.

Each feature is made by a well defined AMQP message using specific subject, annotations and payload in order to bring all MQTT related information like retain flag, will QoS, will topic and so on but using AMQP semantic.

This sort of “specification” doesn’t force to use a specific implementation; the EnMasse project leverages on the Qpid Dispatch Router for connections and Apache Artemis brokers where state is needed (but other implementations could use something different like a simple file system or a database). Of course, some additional services are needed in order to handle LWT and subscriptions (we called them just “Will Service” and “Subscription Service”).

If you are so curious and want to give some feedback on that, you can find all the open source stuff on GitHub in the MQTT over AMQP documentation section.

Feel free to enjoy the related EnMasse implementation ! 😉

Internet of Things : reactive and asynchronous with Vert.x !

vertx_iot

I have to admit … before joining Red Hat I didn’t know about the Eclipse Vert.x project but it took me few days to fall in love with it !

For the other developers who don’t know what Vert.x is, the best definition is …

… a toolkit to build distributed and reactive systems on top of the JVM using an asynchronous non blocking development model

The first big thing is related to develop a reactive system using Vert.x which means :

  • Responsive : the system responds in an acceptable time;
  • Elastic : the system can scale up and scale down;
  • Resilient : the system is designed to handle failures gracefully;
  • Asynchronous : the interaction with the system is achieved using asynchronous messages;

The other big thing is related to use an asynchronous non blocking development model which doesn’t mean to be multi-threading but thanks to the non blocking I/O (i.e. for handling network, file system, …) and callbacks system, it’s possible to handle a huge numbers of events per second using a single thread (aka “event loop”).

You can find a lot of material on the official web site in order to better understand what Vert.x is and all its main features; it’s not my objective to explain it in this very short article that is mostly … you guess … messaging and IoT oriented  🙂

In my opinion, all the above features make Vert.x a great toolkit for building Internet of Things applications where being reactive and asynchronous is a “must” in order to handle millions of connections from devices and all the messages ingested from them.

Vert.x and the Internet of Things

As a toolkit, so made of different components, what are the ones provided by Vert.x and useful to IoT ?

Starting from the Vert.x Core component, there is support for both versions of HTTP protocol so 1.1 and 2.0 in order to develop an HTTP server which can expose a RESTful API to the devices. Today , a lot of web and mobile developers prefer to use this protocol for building their IoT solution leveraging on the deep knowledge they have about the HTTP protocol.

Regarding more IoT oriented protocols, there is the Vert.x MQTT server component which doesn’t provide a full broker but exposes an API that a developer can use in order to handle incoming connections and messages from remote MQTT clients and then building the business logic on top of it, so for example developing a real broker or executing protocol translation (i.e. to/from plain TCP,to/from the Vert.x Event Bus,to/from HTTP,to/from AMQP and so on). The API raises all events related to the connection request from a remote MQTT client and all subsequent incoming messages; at same time, the API provides the way to reply to the remote endpoint. The developer doesn’t need to know how MQTT works on the wire in terms of encoding/decoding messages.

Related to the AMQP 1.0 protocol there are the Vert.x Proton and the AMQP bridge components. The first one provides a thin wrapper around the Apache Qpid Proton engine and can be used for interacting with AMQP based messaging systems as clients (sender and receiver) but even developing a server. The last one provides a bridge between the protocol and the Vert.x Event Bus mostly used for communication between deployed Vert.x verticles. Thanks to this bridge, verticles can interact with AMQP components in a simple way.

Last but not least, the Vert.x Kafka client component which provides access to Apache Kafka for sending and consuming messages from topics and related partitions. A lot of IoT scenarios leverage on Apache Kafka in order to have an ingestion system capable of handling million messages per second.

Conclusion

The current Vert.x code base provides quite interesting components for developing IoT solutions which are already available in the current 3.3.3 version (see Vert.x Proton and AMQP bridge) and that will be available soon in the future 3.4.0 version (see MQTT server and Kafka client). Of course, you don’t need to wait for their official release because, even if under development, you can already adopt these components and provide your feedback to the community.

This ecosystem will grow in the future and Vert.x will be a leading actor in the IoT applications world based on a microservices architecture !

Node-RED : AMQP 1.0 support with the RHEA nodes library

I’m very happy to announce that I have just released the first version of a new messaging and IoT library for providing AMQP 1.0 protocol support on Node-RED that is a very interesting tool for wiring Internet of Things.

This library is based on the official JavaScript RHEA library which provides AMQP 1.0 support for NodeJS applications (developed by one of my best colleagues in Red Hat, Gordon Sim).

Up to now, the available nodes are :

  • sender : for sending messages to an AMQP address
  • receiver : for receiving messages from an AMQP address
  • requester : for executing a request to an AMQP address and waiting a response on the address specified in the “reply-to” field
  • responder : for receiving a request on an AMQP address and sending the response on the address specified in the “reply-to” field

Of course it’s open source (available on GitHub) with a getting started guide and documentation available on the related Wiki.

amqp_nodes

I published it on NPM (Node-RED is completely based on NodeJS) at following link but you can find it even on the official Node-Red web site in the flows section (searching for “amqp” for example).

Enjoying AMQP messaging and IoT with Node-RED flow programming !

A routing IoT gateway to the Cloud

Let’s start with an on-premise solution …

Imagine that you have an embedded solution (or if you like it … an IoT solution) with a bunch of tiny devices which are connected to an on-premise server which receives telemetry data from them and is able to execute some elaboration in order to show information in real time on a dashboard and control the devices.

Imagine that your solution is based on the AMQP protocol and perhaps your on-premise server is running a messaging broker for gathering data from devices as messages through the local network.

Imagine that, due to your very constrained devices, the security in the network is guaranteed only at data level by encrypting the body of every single AMQP message. It’s possible that due to their complexity and need of more resources (CPU and memory) you can’t use sophisticated algorithms (i.e. DES, 3DES, AES, …) on your devices but only simple ones (i.e. TEA, ..).

Your solution is just working great in your environment.

… but now we want to move it to the Cloud

Imagine that for some reasons you need to change the on-premise nature of your solution and you want to connect the devices directly to the cloud with a very strict rule : nothing to change on the devices. At least you can change some configuration parameter (i.e. server ip, …) but not the way and the protocol they are using for communication.

The first simple solution could be moving your messaging broker from the on-premise server on a IaaS in the Cloud; just changing connection parameters on your devices and all continue to work as before.

The big problem now is that your data are sent through the public network and your security is based on a simple encryption algorithm applied only on the payload of the messages. For this reason, you start to think about using SSL/TLS in order to have security at connection level on top of TCP/IP, data encryption and server authentication.

Start to think about it but then … wait … I can’t use SSL/TLS on my tiny devices … they don’t have the needed resources in terms of CPU and memory … and now ?

Fog computing and IoT gateway : the solution ?

You know about “fog computing” (the new buzz word after IoT ?) and that you can solve your problem using an IoT gateway. Having this gateway could mean to have an intelligent piece of software which is able to gather data from the local network, process them in some way and then send them to the Cloud. The gateway could give you more features like filtering on data (sending only part of them), offline handling (if the Cloud isn’t reachable) and complex local processing but … wait … you don’t want it … you just want that data arrives to the Cloud in the same way as before (to the on-premise server) and for now you don’t need other additional great features.

Could we have a very simple IoT gateway with only the two following features we need :

  • SSL/TLS protocol support on behalf of the tiny devices;
  • traffic routing from devices to the Cloud in a transparent way;

The answer is … yes ! You have such solution and it’s provided by the Qpid Dispatch Router project from the ASF (Apache Software Foundation).

I already wrote about it in some previous articles [1] [3] so let me just show how you can use the router in a way that solve your “porting” problem.

The router just needs the right configuration

In order to show in a very simple way how to configure the router for our objective, we can use the Azure IoT Hub as Cloud platform for the IoT. As all the Azure messaging services like Service Bus and Event Hub, the IoT Hub needs an encrypted connection based on the SSL/TLS protocol … so it’s the problem we want to solve for our non SSL capable devices.

For the sake of simplicity we can run the router on a Raspberry Pi using the Raspbian distribution as OS; you can read about installing the Qpid Dispatch Router on Linux and on the Raspberry Pi in these articles [2] [4].

The main point is the configuration needed for the router in order to connect to an IoT Hub and routing the traffic from devices to it.

First of all we have to consider all the addresses that at AMQP level are used in order to send telemetry data to the hub, receive commands and reply with feedback. All these information are deeply explained here [5] [6].

The routing mechanism used in this configuration is the “link routing” [3] which means that the router creates a sort of “tunneling” between devices and the IoT Hub; it opens the TCP/IP connection with the hub, establishing it with SSL/TLS on top, and then opens the AMQP connection. All the SSL/TLS stuff happens between router and IoT Hub and the devices aren’t aware about it. You can see what happens through the router trace :

pi@raspberrypi:~ $ PN_TRACE_FRM=1 qdrouterd --conf ex06_iothub.conf
Sat Jul 23 11:56:17 2016 SERVER (info) Container Name: Router.A
Sat Jul 23 11:56:17 2016 ROUTER (info) Router started in Standalone mode
Sat Jul 23 11:56:17 2016 ROUTER_CORE (info) Router Core thread running. 0/Router.A
Sat Jul 23 11:56:17 2016 ROUTER_CORE (info) In-process subscription M/$management
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$management
Sat Jul 23 11:56:18 2016 AGENT (info) Activating management agent on $_management_internal
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$_management_internal
Sat Jul 23 11:56:18 2016 DISPLAYNAME (info) Activating DisplayNameService on $displayname
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$displayname
Sat Jul 23 11:56:18 2016 CONN_MGR (info) Configured Listener: 0.0.0.0:5672 proto=any role=normal
Listening on 0.0.0.0:5672
Sat Jul 23 11:56:18 2016 CONN_MGR (info) Configured Connector: ppatiernoiothub.azure-devices.net:5671 proto=any role=on-demand
Sat Jul 23 11:56:20 2016 POLICY (info) Policy configured maximumConnections: 0, policyFolder: '', access rules enabled: 'false'
Sat Jul 23 11:56:20 2016 SERVER (info) Operational, 4 Threads Running
Connected to ppatiernoiothub.azure-devices.net:5671
[0x19dc6c8]: -> SASL
[0x19dc6c8]:0 -> @sasl-init(65) [mechanism=:ANONYMOUS, initial-response=b"anonymous@raspberrypi"]
[0x19dc6c8]: -> AMQP
[0x19dc6c8]:0 -> @open(16) [container-id="Router.A", hostname="ppatiernoiothub.azure-devices.net", max-frame-size=65536, channel-max=32767, idle-time-out=60000, offered-capabilities=:"ANONYMOUS-RELAY", properties={:product="qpid-dispatch-router", :version="0.6.0"}]
[0x19dc6c8]: <- SASL
[0x19dc6c8]:0 <- @sasl-mechanisms(64) [sasl-server-mechanisms=@PN_SYMBOL[:EXTERNAL, :MSSBCBS, :ANONYMOUS, :PLAIN]]
[0x19dc6c8]:0 <- @sasl-outcome(68) 
[0x19dc6c8]: <- AMQP
[0x19dc6c8]:0 <- @open(16) [container-id="DeviceGateway_1766cd14067b4c4b8008b15ba75f1fd6", hostname="10.0.0.56", max-frame-size=65536, channel-max=8191, idle-time-out=240000]

At this point, the devices can connect locally to the router and when they asked for all the AMQP links related to the IoT Hub addresses, they will be tunneled by the router : the AMQP “attach” performatives are routed to the IoT Hub through the connection with the router. The communication then continues on this link in terms of message transfers directly between IoT Hub and devices but all encrypted until the router through the SSL/TLS protocol.router_iothub

The router configuration is something like that :

listener {
 addr: 0.0.0.0
 port: 5672
 authenticatePeer: no
}

ssl-profile {
 name: azure-ssl-profile
 cert-db: /opt/qdrouterd/Equifax_Secure_Certificate_Authority.pem
}

connector {
 name: IOTHUB
 addr: <iotHub>.azure-devices.net
 port: 5671
 role: on-demand
 sasl-mechanisms: ANONYMOUS
 ssl-profile: azure-ssl-profile
 idleTimeoutSeconds: 120
}

# sending CBS token
linkRoute {
 prefix: $cbs/
 connection: IOTHUB
 dir: in
}

# receiving the status of CBS token request
linkRoute {
 prefix: $cbs/
 connection: IOTHUB
 dir: out
}

# sending telemetry path and command replies from device to hub on : devices/<DEVICE_ID>/messages/events
# ATTENTION ! Here we need CBS Token
linkRoute {
 prefix: devices/
 connection: IOTHUB
 dir: in
}

# receiving command on device from hub on : devices/<DEVICE_ID>/messages/deviceBound
# ATTENTION ! Here we need CBS Token
linkRoute {
 prefix: devices/
 connection: IOTHUB
 dir: out
}

The main points in the configuration are :

  • a listener entity which defines that the router accept incoming AMQP connections on port 5672 (not encrypted);
  • the ssl-profile entity in order to configure the parameter for SSL/TLS connection to the IoT Hub and specifically the CA certificate to use for server authentication;
  • the connector entity which defines the way the router connects to the IoT Hub (address and port) using the above SSL profile;

After above parameters there is a bunch of linkRoute entities which define what are the addresses that should be link routed by the router from devices to the hub (using the specified connector).

You can find the complete configuration file here.

The Netduino Plus 2 use case

In order to develop an application very quickly on device side I decided to use my knowledge about .Net Micro Framework using a board that hasn’t the SSL/TLS support : the Netduino Plus 2 board.

The simple application is able to send a message to the IoT Hub and receive a new one replying with a feedback. All the code is available here.

In the following pictures you can see the message sent by the board and the command received (with the related feedback) through the Device Explorer tool.

01

02

Conclusion

Of course, the Qpid Dispatch Router project has a greater object than I showed here that could be providing connection to messaging services at scale thanks a more complex router network, with a path redundancy feature to reach a broker or a simple receiver.

In this article, I just showed a different way to use it in order to give more power to tiny devices which aren’t able to connect to AMQP based services due to their limitation (in this case the lack of SSL/TLS support).

If you consider the starting point, the configuration change could be avoided because the router could have same IP address and AMQP listening port as the previous on-premise server .

It means that only adding the router configured for the Cloud connection solves the problem !

[1] Routing AMQP : the Qpid Dispatch Router project

[2] Qpid Dispatch Router installation on your Linux machine

[3] Routing mechanisms for AMQP protocol

[4] My Raspberry Pi runs the Qpid Dispatch Router

[5] Connecting to the Azure IoT Hub using an AMQP stack

[6] Azure IoT Hub : commands and feedback using AMQP .Net Lite