EnMasse

EnMasse and Eclipse Hono ? Messaging and IoT ? I have some events for you !

 

events

Do you want to learn more about the EnMasse project ?

Do you want to learn the same for the Eclipse Hono project ?

Do you want to know more about their relationship and how they simplify the development and deployment of messaging and IoT solutions ?

Well … in the coming weeks my agenda will be packed of events about them. Let’s see !

Eclipse Day Milan 2017

On September 22nd there will be the Eclipse Day Milan 2017 and I’ll be there with the “Eclipse Hono : Connect. Command. Control.” session. What will you see there ?

The open source counterpart to closed and proprietary IoT solutions is called Hono ! Born out of the collaboration of big companies, including Red Hat and Bosch both members of the Eclipse Foundation, Hono is an open source framework which aims to add features, such as device management and authentication, on top of an highly scalable messaging infrastructure in order to guarantee secured data exchange between devices and cloud applications. Using its APIs, devices can send data (telemetry and event) and can be controlled remotely (command/control). During this session we will see the “bricks” that make its architecture, the exposed APIs and the integration with other solutions.

There will be a lot of other great sessions focused on the Eclipse Foundation ecosystem so I think that it will be a really awesome event !

Eclipse IoT Virtual meetup

The Eclipse Foundation hosts a lot of virtual meetups on IoT projects which are part of the foundation itself. Of course, Eclipse Hono is one of them !

On October 11st, I and Kai Hudalla (from Bosch Software Innovations) will have this meetup. We hope to reach a really huge “online” audience in order to show how Hono is a really powerful platform for connecting and control IoT devices at scale.

It will be simpler for you to be there … just take a seat at home !

JavaSI

On October 16th – 17th there will be the JavaSI conference organized by SIOUG.

I and my awesome team mate Ulf Lilleengen will be there with a session and a workshop.

First of all the “EnMasse : open sourcing the messaging and IoT” session as an introduction to the workshop.

Out there there are a lot of “closed” source products for developing messaging and IoT based solutions. What if you want to have more control on your platform ? EnMasse is the answer!

It’s a totally “open” source messaging-as-a-service platform which can be deployed on-premise, in the cloud or even in an hybrid scenario. At the same time it aims to be highly scalable in terms of throughput and the number of connections, based on standard protocols like AMQP 1.0 and MQTT and provides different patterns for messages exchange. Its nature makes EnMasse a great solution for IoT deployments in order to handle millions of connected devices for ingesting telemetry data and controlling them.

Then the “EnMasse – messaging and IoT in your hands” workshop speaking about the EnMasse project and how it’s really great for building messaging and IoT solutions.

Interested in messaging and IoT ? Kubernetes and OpenShift? In this workshop, we will dive into EnMasse, an open source Messaging-as-a-Service platform, built on top of Kubernetes and OpenShift. You will setup OpenShift or Kubernetes, deploy EnMasse, and build an end-2-end solution with edge devices, messaging, and analytics.

You will learn basic Kubernetes and OpenShift concepts as we go, and learn how you can use EnMasse to implement different messaging patterns in your application.

During this workshop you will touch messaging and IoT stuff with your hands !

Advertisements

A lot of fun with … AMQP, Spark, Kafka, EnMasse, MQTT, Vert.x & IoT

When I say to someone that I work for Red Hat they say me “Ah ! Are you working on Linux ?” … No, no, no and … no ! I’m not a Linux guy, I’m not a fan boy but I’m just a daily user 🙂

All people know that Red Hat is THE company which provides the best enterprise Linux distribution well known as Red Hat Enterprise Linux (RHEL) but Red Hat is not only Linux today. Its portfolio is huge : the cloud and containers business with the OpenShift effort, the microservices offer with Vert.x, Wildfly Swarm, Spring Boot, the IoT world with the involvement in the main Eclipse Foundation projects.

The objective of this blog is just showing briefly the projects I worked (or I’m working) on since last year when I was hired on March 1st. They are not “my” projects, they are projects I’m involved because the entire team is working on them … collaboration, you know 🙂

You could be surprised about that but … there is no Linux ! I’m on the messaging & IoT team, so you will see only projects about this stuff 🙂

AMQP – Apache Spark connector

This “little” component is strictly related to the “big” radanalytics.io project which takes the powerful of Apache Spark for analytics (batch, real-time, machine learning, …) running on OpenShift.

Because the messaging team works mainly on projects like ActiveMQ Artemis and the Qpid Dispatch Router, where the main protocol is AMQP 1.0, the idea was developing a connector for Spark Streaming in order to ingest data through this protocol so from queues/topics on a broker or through the router in a direct messaging fashion.

You can find the component here and even an IoT demo here which shows how it’s possible to ingest data through AMQP 1.0 using the EnMasse project (see below) and then executing a real time streaming analytics with Spark Streaming, all running on Kubernetes and OpenShift.

AMQP – Apache Kafka bridge

Apache Kafka is one of the best technologies used today for ingesting data (i.e. IoT related scenarios) with an high throughput. Even in this case, the idea was providing a way for having AMQP 1.0 clients and JMS clients pushing messages to Apache Kafka topics without knowing the related custom protocol.

In this way, if you have such clients because you are already using a broker technology but then you need some specific Kafka features (i.e. re-reading streams), you can just switch the messaging system (from the broker to Kafka) and using the bridge you don’t need to update or modify clients. I showed how this is possible at the Red Hat summit as well and the related demo is available here.

MQTT on EnMasse

EnMasse is an open source messaging platform, with focus on scalability and performance. It can run on your own infrastructure (on premise) or in the cloud, and simplifies the deployment of messaging infrastructure.

It’s based on other open source projects like ActiveMQ Artemis and Qpid Dispatch Router supporting the AMQP 1.0 protocol natively.

In order to provide support for the MQTT protocol, we designed how to take “MQTT over AMQP” so having MQTT features on the AMQP protocol. From the design we moved to develop two main components :

  • the MQTT gateway which handles connections with remote MQTT clients translating all messages from MQTT to AMQP and vice versa;
  • the MQTT LWT (Last and Will Testament) service which provides a way for notifying all clients connected to EnMasse that another client is suddenly died sending them its “will message”. The great thing about this service, is that it works with pure AMQP 1.0 clients so bringing the LWT feature on AMQP as well : for this reason the team is thinking to change its name just in AMQP LWT service.

EnMasse is great for IoT scenarios in order to handle a huge number of connections and ingesting a lot of data using AMQP and MQTT as protocols. I used it in all my IoT demos for showing how it’s possible to integrate it with streaming and analytics frameworks. It’s also the main choice as messaging infrastructure in the cloud for the Eclipse Hono project.

Vert.x and the IoT components

Vert.x is a great toolkit for developing reactive applications running on a JVM.

The reactive applications manifesto fits really well for IoT scenarios where responsiveness, resiliency, elasticity and the communication driven by messages are the pillars of all the IoT solutions.

Starting to work on the MQTT gateway for EnMasse using Vert.x for that, I decided to develop an MQTT server that was just able to handle communication with remote clients providing an API for interacting with them : this component was used for bridging MQTT to AMQP (in EnMasse) but can be used for any scenario where a sort of protocol translation or integration is needed (i.e. MQTT to Vert.x Event Bus, to Kafka, …). Pay attention, it’s not a full broker !

The other component was the Apache Kafka client, mainly developed by Julien Viet (lead on Vert.x) and then passed to me as maintainer for improving it and adding new features from the first release.

Finally, thanks to the Google Summer of Code, during the last 2 months I have been mentoring a student who is working on developing a Vert.x native MQTT client.

As you can see the Vert.x toolkit is really growing from an IoT perspective other then providing a lot of components useful for developing pure microservices based solutions.

Eclipse Hono

Eclipse Hono is a project under the big Eclipse IoT umbrealla in the Eclipse Foundation. It provides a service interfaces for connecting large numbers of IoT devices to a back end and interacting with them in a uniform way regardless of the device communication protocol.

It supports scalable and secure ingestion of large volumes of sensor data by means of its Telemetry API. The Command & Control API allows for sending commands (request messages) to devices and receive a reply to such a command from a device asynchronously in a reliable way.

This project is mainly developed by Red Hat and Bosch and I gave my support on designing all the API other then implementing the MQTT adapter even in this case using the Vert.x MQTT server component.

Because Eclipse Hono works on top of a messaging infrastructure for allowing messages exchange, the main choice was using ActiveMQ Artemis and the Qpid Dispatch Router even running them using Kubernetes and OpenShift with EnMasse.

Apache Kafka

Finally, I was involved to develop a POC named “barnabas” (a messenger character from a Frank Kafka novel :-)) in order to take Apache Kafka running on OpenShift.

Considering the stetaful nature of a project like Kafka, I started when Kubernetes didn’t offer the StatefulSets feature doing something similar by myself. Today, the available deploy is based on StatefulSets and it’s a work in progress on which I’ll continue to work for pushing the POC to the next level.

Apache Kafka is a really great project which has its own use cases in the messaging world; today it’s more powerful thanks to the new Streams API which allows to execute a real time streaming analytics using topics from your cluster and running simple applications. My next step is to move my EnMasse + Spark demo to an EnMasse + Kafka (and streaming) deployment. I’m also giving my support on the Apache Kafka code.

Conclusion

The variety and heterogeneity of all the above projects is giving me a lot of fun in my day by day work even collaborating with different people with different knowledge. I like learning new stuff and the great thing is that … things to learn are endless ! 🙂