Eclipse Hono : “Connect. Command. Control” … even on OpenShift !

The Eclipse Foundation is the main open source community which has a great focus on the Internet of Things world and the related Eclipse IoT ecosystem involves a lot of different projects and partners like Red Hat, Bosch, Eurotech, IBM and many more. Recently, publishing this white paper, they showed a full stack with all the available tools for building a complete IoT solution from devices to the Cloud through the gateways.

selection_005

In relation to the Cloud side, one of the main problems in the IoT world is the ability to handle millions of connections from devices (and gateways) in the field, their registration for managing authentication and authorisation and, last but not least, the ingestion of the data received from them like telemetry or events. Finally, the last point is related to control these devices sending them commands in order to execute actions in the environment around them or upgrading their software and configuration.

The Eclipse Hono™ project is the answer to these problem !

The APIs

From the official web site, we can read :

Eclipse Hono™ provides remote service interfaces for connecting large numbers of IoT devices to a back end and interacting with them in a uniform way regardless of the device communication protocol.

The mantra from the landing page of the web site project is “Connect. Command. Control” which is made a reality through a well defined API for :

  • Registration : for handling the requests related to the registration (so creation) of a new device so that it can be authorised to connect. It’s also possible retrieving information about registered devices or delete them;
  • Telemetry : for the ingestion of a large volume of data from devices (i.e sensors) available for analysis to the backend applications;
  • Event : for receiving specific events (i.e. alarms, notification, …) from devices for making decision on the Cloud side. This API is quite similar to a telemetry path but it uses a “different” channel in order to avoid such events going through the overwhelmed telemetry one;
  • Command & Control : for sending commands to the devices (from a backend application) for executing operations and actions on the field (receiving the related response) and/or upgrading local software and configuration;

All the above APIs are accessible through the AMQP 1.0 protocol (the only standardised AMQP version!) and it means that they are defined in terms of addresses on which devices need to connect for interacting with the system and the properties/content of the messages; of course, it’s true not only for devices but even for business and backend applications which can receive data from devices or send them commands. In any case, it doesn’t mean that devices which aren’t able to speak such protocol can’t connect but they can leverage on the protocol adapters provided by Hono; the current implementation provides an MQTT and HTTP REST adapter.

All these APIs are though in order to allow multi-tenancy so that using a single deployment, it’s possible to handle channels for different tenants so that each of them can’t see data or exchanged messages from the others.

The Architecture

The main components which build the Eclipse Hono™ architecture are :

  1. Protocol Adapters : these components adapt a device protocol to the first citizens protocol used in Hono, the AMQP 1.0. Today, an MQTT and HTT REST adapters are provided out of box but thanks to the available interfaces, the user can develop a new adapter even for some custom protocols;
  2. Hono Server : this is the main component to which devices can connect directly through AMQP 1.0 or through the protocol adapters. It’s in charge to expose the APIs in terms of endpoints and handling the authentication and authorisation of devices;
  3. Qpid Dispatch Router : this is an AMQP 1.0 router, part of the Apache Qpid project, which provides the underlying infrastructure for handling millions of connections from devices in the fields. The simpler deployment can use only one router but in order to guarantee reliability and high volume ingestion, a router network should be used;
  4. ActiveMQ Artemis : this is the broker mainly used for handling command and control API so for storing commands which have to be delivered to devices;

While the devices connect directly to the Hono Server (or through protocol adapters), the backend applications connect to the Qpid Dispatch Router leveraging on direct addressing for receiving telemetry data (if no application is online, no devices are able to send data) or sending commands (the queus backed in the broker are reachable through the router).

selection_004

The running environment

All the artifacts from the project are provided as Docker images for each of the above components that can run using Docker Compose (the Docker Swarm support will be available soon) or using a more focused Cloud platform like OpenShift (compatible with Kubernetes deployment as well).

selection_006

Regarding the OpenShift deployment, the building process of the Hono project provides a bunch of YAML files which describe the objects to deploy like deployment configurations, services, persistent volumes and related claims. All the need is having an OpenShift platform instance accessible and deploy such objects for having Eclipse Hono™ running in a full featured Cloud environment with all the related scaling capabilities.

hono_openshift

The example deployment is based on four pods, one for each of the main components so there is the router pod, the Hono Server pod and one pod for each protocol adapter; of course if you need the command & control path, even the broker pod need to be deployed.

In order to try it, an OpenShift Origin instance can be used on a local PC for deploying the entire Hono solution; for example, the above picture is related to my tests where you can see all the pods running on OpenShift (left side) with simulated devices that interact using MQTT and HTTP REST (on the right side).

So what are you waiting for ? Give it a try !

Conclusion

In my mind every IoT solution should be made of three different layers (a little bit different from the Eclipse vision) : the devices/gateways, the connectivity layer and the service layer.

While the Eclipse Kura project fits in the gateways layer and the Eclipse Kapua in the service layer, Eclipse Hono is the glue between them in order to handle the connections from devices and making available their data to the backend services (and vice versa in the opposite direction for command and control). Thanks to the API standardisation  and the usage of a standard protocol like AMQP 1.0, Hono can be used for connecting any kind of devices with any kind of services; of course leveraging on a these three project has the big advantage of having big companies working on them, mainly Red Hat, Bosch and Eurotech.

Finally, the solution is always the same …. open source and collaboration ! 😉

 

Designing MQTT over AMQP

mqtt_over_amqp

Let’s think about what we can consider some weaknesses of the MQTT protocol …

  • it provides native publish/subscribe pattern only; having request/reply is quite cumbersome with “correlation injection” inside topics and payload messages;
  • it doesn’t provide flow control;
  • it doesn’t have metadata information inside messages (i.e. content type);
  • it’s fully broker centric;

but there are some unique features that we have to consider as strengths for this protocol as well …

  • retain message also known as “last well known” message;
  • last will testament (LWT);
  • session handling;

Now, let’s think about the main features that fill the gap for the MQTT protocol but provided by AMQP …

  • native support for both publish/subscribe and request/reply patterns (correlation for free);
  • flow control at different levels (with max window size and message credits);
  • a full type system and metadata information on each message;
  • peer-to-peer model (the broker is just a peer);

but it lacks of the above three MQTT features !

So how greater could be AMQP protocol having such features on top of it ?

Under the open source EnMasse project, I have been working on having a design (so a kind of “specification”) for having retain message, last will testament and session handling over AMQP. At same time I have been developing an MQTT “gateway” in order to allow remote MQTT clients to connect to an AMQP based “Message as a Service” like EnMasse.

Having such a design means that not only an MQTT client can leverage on receiving retain message after subscribing to a topic, sending its LWT on connection or receiving messages for a topic when it was offline; it means having the above features for native AMQP clients as well.

Each feature is made by a well defined AMQP message using specific subject, annotations and payload in order to bring all MQTT related information like retain flag, will QoS, will topic and so on but using AMQP semantic.

This sort of “specification” doesn’t force to use a specific implementation; the EnMasse project leverages on the Qpid Dispatch Router for connections and Apache Artemis brokers where state is needed (but other implementations could use something different like a simple file system or a database). Of course, some additional services are needed in order to handle LWT and subscriptions (we called them just “Will Service” and “Subscription Service”).

If you are so curious and want to give some feedback on that, you can find all the open source stuff on GitHub in the MQTT over AMQP documentation section.

Feel free to enjoy the related EnMasse implementation ! 😉

Internet of Things : reactive and asynchronous with Vert.x !

vertx_iot

I have to admit … before joining Red Hat I didn’t know about the Eclipse Vert.x project but it took me few days to fall in love with it !

For the other developers who don’t know what Vert.x is, the best definition is …

… a toolkit to build distributed and reactive systems on top of the JVM using an asynchronous non blocking development model

The first big thing is related to develop a reactive system using Vert.x which means :

  • Responsive : the system responds in an acceptable time;
  • Elastic : the system can scale up and scale down;
  • Resilient : the system is designed to handle failures gracefully;
  • Asynchronous : the interaction with the system is achieved using asynchronous messages;

The other big thing is related to use an asynchronous non blocking development model which doesn’t mean to be multi-threading but thanks to the non blocking I/O (i.e. for handling network, file system, …) and callbacks system, it’s possible to handle a huge numbers of events per second using a single thread (aka “event loop”).

You can find a lot of material on the official web site in order to better understand what Vert.x is and all its main features; it’s not my objective to explain it in this very short article that is mostly … you guess … messaging and IoT oriented  🙂

In my opinion, all the above features make Vert.x a great toolkit for building Internet of Things applications where being reactive and asynchronous is a “must” in order to handle millions of connections from devices and all the messages ingested from them.

Vert.x and the Internet of Things

As a toolkit, so made of different components, what are the ones provided by Vert.x and useful to IoT ?

Starting from the Vert.x Core component, there is support for both versions of HTTP protocol so 1.1 and 2.0 in order to develop an HTTP server which can expose a RESTful API to the devices. Today , a lot of web and mobile developers prefer to use this protocol for building their IoT solution leveraging on the deep knowledge they have about the HTTP protocol.

Regarding more IoT oriented protocols, there is the Vert.x MQTT server component which doesn’t provide a full broker but exposes an API that a developer can use in order to handle incoming connections and messages from remote MQTT clients and then building the business logic on top of it, so for example developing a real broker or executing protocol translation (i.e. to/from plain TCP,to/from the Vert.x Event Bus,to/from HTTP,to/from AMQP and so on). The API raises all events related to the connection request from a remote MQTT client and all subsequent incoming messages; at same time, the API provides the way to reply to the remote endpoint. The developer doesn’t need to know how MQTT works on the wire in terms of encoding/decoding messages.

Related to the AMQP 1.0 protocol there are the Vert.x Proton and the AMQP bridge components. The first one provides a thin wrapper around the Apache Qpid Proton engine and can be used for interacting with AMQP based messaging systems as clients (sender and receiver) but even developing a server. The last one provides a bridge between the protocol and the Vert.x Event Bus mostly used for communication between deployed Vert.x verticles. Thanks to this bridge, verticles can interact with AMQP components in a simple way.

Last but not least, the Vert.x Kafka client component which provides access to Apache Kafka for sending and consuming messages from topics and related partitions. A lot of IoT scenarios leverage on Apache Kafka in order to have an ingestion system capable of handling million messages per second.

Conclusion

The current Vert.x code base provides quite interesting components for developing IoT solutions which are already available in the current 3.3.3 version (see Vert.x Proton and AMQP bridge) and that will be available soon in the future 3.4.0 version (see MQTT server and Kafka client). Of course, you don’t need to wait for their official release because, even if under development, you can already adopt these components and provide your feedback to the community.

This ecosystem will grow in the future and Vert.x will be a leading actor in the IoT applications world based on a microservices architecture !

The power of collaboration and open source

Waiting for the flight coming back home … here in Stuttgart after a great Eclipse IoT Day during the EclipseCon conference in Ludwigsburg … thinking about the power of collaboration and open source …

Eclipse IoT : a big announcement

Few days ago, the Eclipse Foundation announced a collaboration between three big companies for developing the “Internet of Things” open source platform of the future under the Eclipse IoT umbrella : Red Hat, Bosch and Eurotech.

Yesterday, most of the Eclipse IoT Day sessions were focused on the projects that these companies are leading in order to build such a platform; three names that you should remember … Kura, Hono and Kapua.

From devices and gateway on the field … through the IoT connectivity at scale … to cloud services for gathering insights from data and controlling devices. You can find a lot of information about these projects on the related official web sites and public repositories so it’s not my intention digging into them in this post but just sharing my impressions about the power of collaboration and open source.

Let me just share a picture of the vision that Red Hat has about this IoT platform involving a broader ecosystem of open source projects not only from Eclipse Foundation but even from Apache Foundation.

red_hat_iot_eclipsecon

For sure you have Eclipse IoT projects but even the Apache Qpid Dispatch Router for connectivity and messaging at scale thanks to an AMQP router network, Vert.x based microservices for handling devices connectivity and protocols translation, ActiveMQ Artemis broker for the “store and forward” needs, AMQP – Spark Streaming integration for real time analytics and finally a future Apache Kafka support.

All the best open source projects for a great IoT platform !

From closed source to … open sourcing and collaboration

For a long time I worked in a company with closed source products using … closed source products. In the spare time, I decided to start sharing my knowledge and improving it developing open source software and … I have to say … my life is changed.

From that time I gave something to the community but I received ten times what I have done. Thanks to open sourcing what I was doing, my knowledge is increased exponentially because developers all around the world were able to see my code giving me suggestions and improvements … so then the power over collaboration.

Red Hat : the Open Organization

In the world … there is only one big company based on these pillars … Red Hat !

Today, I’m so proud to be part of such a company where people can use their creativity and having fun during their work days for building open source solutions collaborating with the best developers you can ever imagine. Thanks to the “remotees”, this potential grows exponentially because talented developers can be all around the world and Red Hat searches for them every day. You can imagine that in such a context … collaboration is the pillar !

Periodically … for meetings or conferences … you have the chance to meet your colleagues in person discussing about projects you (together) are working on but even …. drinking a beer and sharing experiences … another time … collaboration !

This week was even the “We Are Red Hat Week” (WARHW … a very complex acronym). Every year a lot of activities are organized in the Red Hat offices all around the world for team building and enjoying different experiences other than working. Here in Ludwigsburg we decided to take a picture at least !

warhw

A lot of companies should take the Red Hat model … the open source and collaboration won !

Linux Day 2016 : Speaking about IoT … on the scene of the crime !

linux_day_2016

After about 10 years, Saturday October 22nd, I’ll come back on the scene of the crime !

What ? You could ask …

Next Saturday will be the Linux Day … something like a national day, here in Italy, to celebrate the open source software and all the ecosystem around it. Thanks to the NaLUG regional community, I’ll be speaker with the session “The Internet of Things … Babel” speaking about the main patterns but mostly the main protocols that are available in the IoT space.

nalug_be439_450x450

The conference will take place at University “Federico II” in Naples where I graduated in the 2006; for this reason I’ll come back on the scene of the crime but this time on the opposite side ! I’m very excited about that !

Of course, it’s a pleasure for me helping NaLUG community and I’d like to thank them for accepting my session proposal.

You can find all the details and the agenda at this link.

Come and join us !

Node-RED : AMQP 1.0 support with the RHEA nodes library

I’m very happy to announce that I have just released the first version of a new messaging and IoT library for providing AMQP 1.0 protocol support on Node-RED that is a very interesting tool for wiring Internet of Things.

This library is based on the official JavaScript RHEA library which provides AMQP 1.0 support for NodeJS applications (developed by one of my best colleagues in Red Hat, Gordon Sim).

Up to now, the available nodes are :

  • sender : for sending messages to an AMQP address
  • receiver : for receiving messages from an AMQP address
  • requester : for executing a request to an AMQP address and waiting a response on the address specified in the “reply-to” field
  • responder : for receiving a request on an AMQP address and sending the response on the address specified in the “reply-to” field

Of course it’s open source (available on GitHub) with a getting started guide and documentation available on the related Wiki.

amqp_nodes

I published it on NPM (Node-RED is completely based on NodeJS) at following link but you can find it even on the official Node-Red web site in the flows section (searching for “amqp” for example).

Enjoying AMQP messaging and IoT with Node-RED flow programming !

Intel Joule debate and … #thesixthsense on #IoT

After Intel Developer Forum 2016 (IDF) with the new Joule IoT Dev Kit announcement, a debate started between some colleagues here in Red Hat on using this kind of platforms only for maker projects or in the enterprise world as well.

This debate became a post on the Red Hat Developers blog titled “Intel Joule Debate: A maker platform ready for widespread IoT use?”.

It was funny because before it was published, I started a #thesixthsense #IoT series on Twitter worried about how the embedded systems “could” be in a near future using high level tools by high level developers …

thesixthsense_iot

I think that the blog post worths a read and I hope you’ll enjoy it !