Designing MQTT over AMQP

mqtt_over_amqp

Let’s think about what we can consider some weaknesses of the MQTT protocol …

  • it provides native publish/subscribe pattern only; having request/reply is quite cumbersome with “correlation injection” inside topics and payload messages;
  • it doesn’t provide flow control;
  • it doesn’t have metadata information inside messages (i.e. content type);
  • it’s fully broker centric;

but there are some unique features that we have to consider as strengths for this protocol as well …

  • retain message also known as “last well known” message;
  • last will testament (LWT);
  • session handling;

Now, let’s think about the main features that fill the gap for the MQTT protocol but provided by AMQP …

  • native support for both publish/subscribe and request/reply patterns (correlation for free);
  • flow control at different levels (with max window size and message credits);
  • a full type system and metadata information on each message;
  • peer-to-peer model (the broker is just a peer);

but it lacks of the above three MQTT features !

So how greater could be AMQP protocol having such features on top of it ?

Under the open source EnMasse project, I have been working on having a design (so a kind of “specification”) for having retain message, last will testament and session handling over AMQP. At same time I have been developing an MQTT “gateway” in order to allow remote MQTT clients to connect to an AMQP based “Message as a Service” like EnMasse.

Having such a design means that not only an MQTT client can leverage on receiving retain message after subscribing to a topic, sending its LWT on connection or receiving messages for a topic when it was offline; it means having the above features for native AMQP clients as well.

Each feature is made by a well defined AMQP message using specific subject, annotations and payload in order to bring all MQTT related information like retain flag, will QoS, will topic and so on but using AMQP semantic.

This sort of “specification” doesn’t force to use a specific implementation; the EnMasse project leverages on the Qpid Dispatch Router for connections and Apache Artemis brokers where state is needed (but other implementations could use something different like a simple file system or a database). Of course, some additional services are needed in order to handle LWT and subscriptions (we called them just “Will Service” and “Subscription Service”).

If you are so curious and want to give some feedback on that, you can find all the open source stuff on GitHub in the MQTT over AMQP documentation section.

Feel free to enjoy the related EnMasse implementation ! 😉

Internet of Things : reactive and asynchronous with Vert.x !

vertx_iot

I have to admit … before joining Red Hat I didn’t know about the Eclipse Vert.x project but it took me few days to fall in love with it !

For the other developers who don’t know what Vert.x is, the best definition is …

… a toolkit to build distributed and reactive systems on top of the JVM using an asynchronous non blocking development model

The first big thing is related to develop a reactive system using Vert.x which means :

  • Responsive : the system responds in an acceptable time;
  • Elastic : the system can scale up and scale down;
  • Resilient : the system is designed to handle failures gracefully;
  • Asynchronous : the interaction with the system is achieved using asynchronous messages;

The other big thing is related to use an asynchronous non blocking development model which doesn’t mean to be multi-threading but thanks to the non blocking I/O (i.e. for handling network, file system, …) and callbacks system, it’s possible to handle a huge numbers of events per second using a single thread (aka “event loop”).

You can find a lot of material on the official web site in order to better understand what Vert.x is and all its main features; it’s not my objective to explain it in this very short article that is mostly … you guess … messaging and IoT oriented  🙂

In my opinion, all the above features make Vert.x a great toolkit for building Internet of Things applications where being reactive and asynchronous is a “must” in order to handle millions of connections from devices and all the messages ingested from them.

Vert.x and the Internet of Things

As a toolkit, so made of different components, what are the ones provided by Vert.x and useful to IoT ?

Starting from the Vert.x Core component, there is support for both versions of HTTP protocol so 1.1 and 2.0 in order to develop an HTTP server which can expose a RESTful API to the devices. Today , a lot of web and mobile developers prefer to use this protocol for building their IoT solution leveraging on the deep knowledge they have about the HTTP protocol.

Regarding more IoT oriented protocols, there is the Vert.x MQTT server component which doesn’t provide a full broker but exposes an API that a developer can use in order to handle incoming connections and messages from remote MQTT clients and then building the business logic on top of it, so for example developing a real broker or executing protocol translation (i.e. to/from plain TCP,to/from the Vert.x Event Bus,to/from HTTP,to/from AMQP and so on). The API raises all events related to the connection request from a remote MQTT client and all subsequent incoming messages; at same time, the API provides the way to reply to the remote endpoint. The developer doesn’t need to know how MQTT works on the wire in terms of encoding/decoding messages.

Related to the AMQP 1.0 protocol there are the Vert.x Proton and the AMQP bridge components. The first one provides a thin wrapper around the Apache Qpid Proton engine and can be used for interacting with AMQP based messaging systems as clients (sender and receiver) but even developing a server. The last one provides a bridge between the protocol and the Vert.x Event Bus mostly used for communication between deployed Vert.x verticles. Thanks to this bridge, verticles can interact with AMQP components in a simple way.

Last but not least, the Vert.x Kafka client component which provides access to Apache Kafka for sending and consuming messages from topics and related partitions. A lot of IoT scenarios leverage on Apache Kafka in order to have an ingestion system capable of handling million messages per second.

Conclusion

The current Vert.x code base provides quite interesting components for developing IoT solutions which are already available in the current 3.3.3 version (see Vert.x Proton and AMQP bridge) and that will be available soon in the future 3.4.0 version (see MQTT server and Kafka client). Of course, you don’t need to wait for their official release because, even if under development, you can already adopt these components and provide your feedback to the community.

This ecosystem will grow in the future and Vert.x will be a leading actor in the IoT applications world based on a microservices architecture !

The power of collaboration and open source

Waiting for the flight coming back home … here in Stuttgart after a great Eclipse IoT Day during the EclipseCon conference in Ludwigsburg … thinking about the power of collaboration and open source …

Eclipse IoT : a big announcement

Few days ago, the Eclipse Foundation announced a collaboration between three big companies for developing the “Internet of Things” open source platform of the future under the Eclipse IoT umbrella : Red Hat, Bosch and Eurotech.

Yesterday, most of the Eclipse IoT Day sessions were focused on the projects that these companies are leading in order to build such a platform; three names that you should remember … Kura, Hono and Kapua.

From devices and gateway on the field … through the IoT connectivity at scale … to cloud services for gathering insights from data and controlling devices. You can find a lot of information about these projects on the related official web sites and public repositories so it’s not my intention digging into them in this post but just sharing my impressions about the power of collaboration and open source.

Let me just share a picture of the vision that Red Hat has about this IoT platform involving a broader ecosystem of open source projects not only from Eclipse Foundation but even from Apache Foundation.

red_hat_iot_eclipsecon

For sure you have Eclipse IoT projects but even the Apache Qpid Dispatch Router for connectivity and messaging at scale thanks to an AMQP router network, Vert.x based microservices for handling devices connectivity and protocols translation, ActiveMQ Artemis broker for the “store and forward” needs, AMQP – Spark Streaming integration for real time analytics and finally a future Apache Kafka support.

All the best open source projects for a great IoT platform !

From closed source to … open sourcing and collaboration

For a long time I worked in a company with closed source products using … closed source products. In the spare time, I decided to start sharing my knowledge and improving it developing open source software and … I have to say … my life is changed.

From that time I gave something to the community but I received ten times what I have done. Thanks to open sourcing what I was doing, my knowledge is increased exponentially because developers all around the world were able to see my code giving me suggestions and improvements … so then the power over collaboration.

Red Hat : the Open Organization

In the world … there is only one big company based on these pillars … Red Hat !

Today, I’m so proud to be part of such a company where people can use their creativity and having fun during their work days for building open source solutions collaborating with the best developers you can ever imagine. Thanks to the “remotees”, this potential grows exponentially because talented developers can be all around the world and Red Hat searches for them every day. You can imagine that in such a context … collaboration is the pillar !

Periodically … for meetings or conferences … you have the chance to meet your colleagues in person discussing about projects you (together) are working on but even …. drinking a beer and sharing experiences … another time … collaboration !

This week was even the “We Are Red Hat Week” (WARHW … a very complex acronym). Every year a lot of activities are organized in the Red Hat offices all around the world for team building and enjoying different experiences other than working. Here in Ludwigsburg we decided to take a picture at least !

warhw

A lot of companies should take the Red Hat model … the open source and collaboration won !

Linux Day 2016 : Speaking about IoT … on the scene of the crime !

linux_day_2016

After about 10 years, Saturday October 22nd, I’ll come back on the scene of the crime !

What ? You could ask …

Next Saturday will be the Linux Day … something like a national day, here in Italy, to celebrate the open source software and all the ecosystem around it. Thanks to the NaLUG regional community, I’ll be speaker with the session “The Internet of Things … Babel” speaking about the main patterns but mostly the main protocols that are available in the IoT space.

nalug_be439_450x450

The conference will take place at University “Federico II” in Naples where I graduated in the 2006; for this reason I’ll come back on the scene of the crime but this time on the opposite side ! I’m very excited about that !

Of course, it’s a pleasure for me helping NaLUG community and I’d like to thank them for accepting my session proposal.

You can find all the details and the agenda at this link.

Come and join us !

Node-RED : AMQP 1.0 support with the RHEA nodes library

I’m very happy to announce that I have just released the first version of a new messaging and IoT library for providing AMQP 1.0 protocol support on Node-RED that is a very interesting tool for wiring Internet of Things.

This library is based on the official JavaScript RHEA library which provides AMQP 1.0 support for NodeJS applications (developed by one of my best colleagues in Red Hat, Gordon Sim).

Up to now, the available nodes are :

  • sender : for sending messages to an AMQP address
  • receiver : for receiving messages from an AMQP address
  • requester : for executing a request to an AMQP address and waiting a response on the address specified in the “reply-to” field
  • responder : for receiving a request on an AMQP address and sending the response on the address specified in the “reply-to” field

Of course it’s open source (available on GitHub) with a getting started guide and documentation available on the related Wiki.

amqp_nodes

I published it on NPM (Node-RED is completely based on NodeJS) at following link but you can find it even on the official Node-Red web site in the flows section (searching for “amqp” for example).

Enjoying AMQP messaging and IoT with Node-RED flow programming !

Intel Joule debate and … #thesixthsense on #IoT

After Intel Developer Forum 2016 (IDF) with the new Joule IoT Dev Kit announcement, a debate started between some colleagues here in Red Hat on using this kind of platforms only for maker projects or in the enterprise world as well.

This debate became a post on the Red Hat Developers blog titled “Intel Joule Debate: A maker platform ready for widespread IoT use?”.

It was funny because before it was published, I started a #thesixthsense #IoT series on Twitter worried about how the embedded systems “could” be in a near future using high level tools by high level developers …

thesixthsense_iot

I think that the blog post worths a read and I hope you’ll enjoy it !

Open source nature and community for bringing .Net Micro Framework to a new life !

Yesterday, the biggest manufacturer of .Net Gadgeteer devices announced a bad news to the community: they won’t produce the boards based on it anymore.

To be precise, they won’t abandoned the entire .Net Micro Framework development but “only” all the boards known for the simplicity of developing prototypes avoiding soldering but using simple socket instead.

Of course, I’m speaking about the GHI Electronics.

I have some of their great boards used in a lot of demos and examples on .Net Micro Framework development for the Internet of Things during my sessions around Italy. Today, I can understand their decision: .Net Gadgeteer was created by Microsoft but today there isn’t so much effort on it and the community isn’t so huge for bringing the framework to the next level. In the last years, GHI Electronics was alone in order to support this framework and having it alive. From my point of view, it’s a pity because using .Net Gadgeteer boards, the ideas become reality in a very short time: from the advantage of using socket connection for the hardware to the application development with C# and .Net Micro Framework. The way to a final product is very long … but as starting point it was good.

To this bad news, we can add the silence by SecretLabs about another well-known board in the .Net Micro Framework world: the Netduino.

Netduino was my first love starting from its first version (without Ethernet) to the latest WiFi board. I played a lot with it, learning .Net Micro Framework development, even without the simplicity of hardware connections because I like “flying wires”. Even SecretLabs tried to do something like .Net Gadgeteer with Netduino Go without so much luck (they used a completely different “standard” for that).

The bad news and the silence let me start thinking about the future of the .Net Micro Framework.

Of course, to be precise, the framework life isn’t related to the future of the maker platforms provided by GHI Electronics and SecretLabs, because some companies are designing their own boards for running it.

It was born in the Microsoft research laboratories as the SPOT (Smart Personal Objects Technology) framework presented in the 2002. It became .Net Micro Framework and after some years Microsoft decided to provide it as open source without focusing so much on its development. Developers started to use it but only for hobbystic projects; only few companies decided to start developing real world products.

With the new IoT business, a couple of years ago Microsoft re-started to focus on it showing a lot of demos all around the world (at Build as well) based on .Net Micro Framework devices connected to Azure services (even before IoT Hub but using Service Bus and Event Hub). It seems that today this strong support is ended … Microsoft wants to provide better Cloud experience (the “I” in the IoT business) for all kind of devices without any difference on the running operating system or framework.

It seems to be another bad news for developers who believe in the .Net Micro Framework but fortunately … it’s always open source and the community can change its destiny.

As an open source developers (you know, I work for Red Hat … the company leader on open source) I believe in community projects : I started to write a lot of open sourced code for .Net Micro Framework a bunch of years ago and the community helped me to improve my libraries.

Today, Microsoft is providing few “human” resources to the .Net Micro Framework project hosted on GitHub in order to make the big changes on it and coordinate the community: I agree on that … now the community has the framework in its hands. At same time I know the Microsoft guys who are supporting it … they are great guys … trust me !

I know a lot of great guys who are working on .Net Micro Framework for their commercial products and for this reason I’m confident in the future. I know that the community around it isn’t so huge and it could be a problem for having a platform used as primary choice for embedded development. It’s also true that embedded world (yes ! embedded … not IoT !) is so much heterogeneous and each developer has its preferences: there is space for all of them, from ARM mbed platform to the .Net Micro Framework itself for example.

In the last months the interest about .Net Micro Framework was revamped by the new LILIUM project; to be precise it’s not the new version of the framework which has its own roadmap but a new way to develop in C# and UWP on embedded devices with the power of pre-compiled native code !

In order to define the right way for the community effort, two interesting discussions were opened on GitHub and all the main protagonists are arguing on the objectives and the roadmap: this is the essence of “to be a community”.

The first one is focused on the future of the .Net Micro Framework and its version 5 (today we are at 4.4) and the other one on the differences with LILIUM project; it’s worth for you to follow them both.

Let’s see … the IoT business opened a lot of scenarios … no doors are closed … can the open source nature and the community bring embedded C# development to a new life ?