Uncategorized

Eclipse Hono : Virtual IoT meetup

Virtual IoT - Hono

Yesterday, thanks to the Eclipse Foundation I had the chance to talk about Eclipse Hono as speaker for this virtual IoT meetup as part of a meetup series where the focus is on the Eclipse IoT projects. I was with Kai Hudalla (Chief Software Architect at BoschSI) who is co-lead and main contributor on Hono.

It was my first virtual meetup and a really exciting experience for me with almost 90 “virtual” attendees and a lot of interesting questions showing the interest that developers had about this “new” project.

If you didn’t have a chance to watch the session or you want to re-watch it, you can find the recording on YouTube; the slides deck is available here as well.

Advertisements

Eclipse Day Milan 2017 : speaking about Eclipse Hono !

Last Friday, September 22nd, I was in Milan for the first Eclipse Day !

It was a really great event with a good number of attendees (there were about 100 people) following sessions from italian and international speakers.

I had my session speaking about Eclipse Hono, digging into its architecture, its API and what it provides today for building IoT solutions. Of course, having an open source platform for the Internet of Things gets a lot of interest from developers.

Being in Milan gave me the chance to meet some colleagues from the Red Hat office as well 🙂

You can find the slides about my session here.

 eclipseday_02 eclipseday_03

eclipseday_04

eclipseday_05 eclipseday_06

 

GnatMQ : a more community driven future !

When I developed the M2Mqtt library, a lot of people asked me to develop a .Net based MQTT broker as well. It wasn’t my initial intention but at same time I thought that it could be a good idea trying to reuse part of the M2Mqtt client in order to do that : this is the way how GnatMQ was born.

From the beginning I said that it wasn’t a production ready MQTT broker with decent performance for its job but in the last years it turned out that a lot of developers started to use it for their IoT solutions.

Due to my last really busy year and pushed by this issue opened in the GnatMQ repo, this week I decided to create a GnatMQ organization and making this broker more open source then before. I added some developers who asked me to be member of such organization so that they can drive the GnatMQ future better than me : this members proved to contribute to the project in the past pushing PRs on it. Of course, I won’t give up on that, I will always feel like the project’s father and I’ll coordinate such community.

So … it means that GnatMQ isn’t dead … the community and the power of collaboration will move it to the next level, maybe having a really good fully .Net based MQTT broker !

 

EnMasse and Eclipse Hono ? Messaging and IoT ? I have some events for you !

 

events

Do you want to learn more about the EnMasse project ?

Do you want to learn the same for the Eclipse Hono project ?

Do you want to know more about their relationship and how they simplify the development and deployment of messaging and IoT solutions ?

Well … in the coming weeks my agenda will be packed of events about them. Let’s see !

Eclipse Day Milan 2017

On September 22nd there will be the Eclipse Day Milan 2017 and I’ll be there with the “Eclipse Hono : Connect. Command. Control.” session. What will you see there ?

The open source counterpart to closed and proprietary IoT solutions is called Hono ! Born out of the collaboration of big companies, including Red Hat and Bosch both members of the Eclipse Foundation, Hono is an open source framework which aims to add features, such as device management and authentication, on top of an highly scalable messaging infrastructure in order to guarantee secured data exchange between devices and cloud applications. Using its APIs, devices can send data (telemetry and event) and can be controlled remotely (command/control). During this session we will see the “bricks” that make its architecture, the exposed APIs and the integration with other solutions.

There will be a lot of other great sessions focused on the Eclipse Foundation ecosystem so I think that it will be a really awesome event !

Eclipse IoT Virtual meetup

The Eclipse Foundation hosts a lot of virtual meetups on IoT projects which are part of the foundation itself. Of course, Eclipse Hono is one of them !

On October 11st, I and Kai Hudalla (from Bosch Software Innovations) will have this meetup. We hope to reach a really huge “online” audience in order to show how Hono is a really powerful platform for connecting and control IoT devices at scale.

It will be simpler for you to be there … just take a seat at home !

JavaSI

On October 16th – 17th there will be the JavaSI conference organized by SIOUG.

I and my awesome team mate Ulf Lilleengen will be there with a session and a workshop.

First of all the “EnMasse : open sourcing the messaging and IoT” session as an introduction to the workshop.

Out there there are a lot of “closed” source products for developing messaging and IoT based solutions. What if you want to have more control on your platform ? EnMasse is the answer!

It’s a totally “open” source messaging-as-a-service platform which can be deployed on-premise, in the cloud or even in an hybrid scenario. At the same time it aims to be highly scalable in terms of throughput and the number of connections, based on standard protocols like AMQP 1.0 and MQTT and provides different patterns for messages exchange. Its nature makes EnMasse a great solution for IoT deployments in order to handle millions of connected devices for ingesting telemetry data and controlling them.

Then the “EnMasse – messaging and IoT in your hands” workshop speaking about the EnMasse project and how it’s really great for building messaging and IoT solutions.

Interested in messaging and IoT ? Kubernetes and OpenShift? In this workshop, we will dive into EnMasse, an open source Messaging-as-a-Service platform, built on top of Kubernetes and OpenShift. You will setup OpenShift or Kubernetes, deploy EnMasse, and build an end-2-end solution with edge devices, messaging, and analytics.

You will learn basic Kubernetes and OpenShift concepts as we go, and learn how you can use EnMasse to implement different messaging patterns in your application.

During this workshop you will touch messaging and IoT stuff with your hands !

MQTT v5 : what is on the way ?

“MQTT is a lightweight protocol for IoT” …. “MQTT lacks a lot of features” … and how many of other sentences you have heard speaking about MQTT with others developers ?

During the last year, the OASIS committee has worked a lot on the new MQTT v5 specification pushing the protocol to the next level in both directions : a lot of new features are coming and they will fill (part of) the gap that it has against other protocols which already provide them (my opinion is that, from some points of view, the new MQTT v5 is more AMQP-ish 🙂 ); on the other side, don’t tell me that MQTT is lightweight as before. Adding features means adding complexity making it heavier and maybe this is the reason why, today, a lot of IoT developers decide to not use AMQP for their projects … but repeating myself … more features mean more complexity and they are very welcome.

By the way, this blog post is about MQTT v5 and the new specification so … let’s start !

(the OASIS committee has opened a publish review and it will be opened until September 8th, you can find more information here if you want to read the entire specification. There is also a “Big ideas for MQTT v5” interesting document here with links to related issues on OASIS web site).

Why from 3.1.1 to 5 ?

A lot of people ask me why this “jump” from 3.1.1 to 5 ! The answer is in the protocol itself !

The CONNECT packet, which brings the connection information from the client to the broker, has a “protocol version” byte inside the variable header : it’s a single byte which provides the revision level of the protocol used by the client. With version 3.1 it was 3 then, moving to the current 3.1.1, it became 4. Starting to write the new specification, the committee decided to align the “marketing” version of the protocol with the “internal” representation on the wire : from 3.1.1 to 5 … so from “protocol version” 4 to 5 !

You can see it even as a really “huge” specification change as it really is in terms of new features.

Properties … not only payload

The “variable header” is changed and now it contains some properties and each property is defined as a key-value pair. Some property are fixed and used in specific packets like for example the “content-type” which describes the type of content in the payload (JSON, XML, …) and the “response topic” used in the new supported request/response pattern (as we’ll see in the next paragraphs). There is the possibility to add “user properties” as well so that the developer can add more key-value pairs for bringing values meaningful at application level : it’s an interesting feature because in some IoT solutions, it could be interesting not sending the payload at all but just values using properties. This aspect is confirmed by the fact that the payload for the PUBLISH message is defined as “optional” now while it’s “required” in the current 3.1.1 specification.

AMQP already had this kind of feature : system properties (i.e. content-type, reply-to, ttl, …) and application properties.

Error handling : now I know what’s really happened

One of the missing thing in the current 3.1.1 specification is the support for a proper “error handling” at application level : when there is an error, the server just shuts down the connection and the client doesn’t have any possibility to know the reason. In the new specification quite much all the packets have a single byte “reason code” as part of the “variable header”.

Alongside the “reason code” there is a “reason string” that can be used for providing a more human readable information about the error.

Such a feature is something that HTTP and AMQP already provided.

Flow control for QoS 1 and 2

Flow control is the main lacking feature in the current 3.1.1 specification … something that the AMQP protocol already had even at different levels (i.e. session window and credits on messages).

The new v5 specification adds a simple flow control based on the “receive maximum” property. With this property the client can limit the number of QoS 1 and 2 messages that it is willing to process concurrently : it defines a limit quota about the number of PUBLISH messages which can be sent without receiving the acknowledge. There is no flow control for QoS 0 messages because as we know there is no acknowledgement mechanism for that; the acknowledgment mechanism for QoS 1 and 2 is used by the server for avoiding sending messages to the client; so overwhelming a client with QoS 0 publications is still possible.

Request/Response pattern … here we are !

The MQTT protocol is well known for its publish/subscribe nature without any built-in support for request/response pattern. With 3.1.1 specification, there is no “standard” way for a requester to specify the topic where it expects to receive a response from a responder : it’s something that could be encoded inside the message payload. The new v5 introduces the “response topic” property (something that AMQP already had with the “reply-to” system property) : using such property, the requester has a “standard” way to specify the subscribed topic on which it expects replies from a responder.

Shared subscriptions

The normal way to work for MQTT is that the broker delivers a received message to ALL subscribers for the topic on which the message is published : today we can call them “non shared” subscriptions. The v5 specification adds the “shared subscription” concept : a subscription can be shared among different clients and the messages load will be spread across them. It means that the broker doesn’t just send the received message to all subscribers but to only one of them. Effectively, the clients are something like “competing consumers” on the shared subscription.

A shared subscription is identified using a special topic filter with following format :

$share/{ShareName}/{filter}

Where :

  • $share is needed for specifying that the subscription is shared
  • {ShareName} is the name of the subscription used for grouping clients (it sounds to me something like the “consumer group” in Apache Kafka)
  • {filter} is the topic filter and it’s already well known for “non shared” subscription

For example, imagine to have a publisher P sending messages on topic “/foo” and we have two different applications A1 and A2 which need to get messages published on this topic for executing different actions (i.e. monitoring, logging, …). Using “non shared” subscriptions we can just have A1 and A2 subscribing to the topic “/foo” and starting to receive messages from that. If the load on the topic increase and we want to leverage on the huge potential we have with a cloud native and containerized applications so that we could spread the load across multiple instances of applications A1 and A2, we can use the “shared” subscriptions in the following way.

From the single topic “/foo” we can move to have :

  • $share/A1/foo
  • $share/A2/foo

All the instances of application A1 can subscribe to the first subscription and the instances of A2 can subscribe to the second one.

In this way, the A1 instances are competing consumers for messages published on “/foo” and the same for A2 instances. We still have all messages published to both applications but the load is spread across different instances (of the same application) thanks to the “shared” subscription support.

Session management

With MQTT a session is represented by the subscriptions for a client and any queued messages (when the client isn’t online). In the 3.1.1 specification, the “clean session” flag is used by the client for specifiying that : the server would delete any existing session and would not save the new session (if set 1); the server would need to recover any existing session on client re-connection (if set 0) and save it on disconnection.

In the new v5, the behavior is changed. First of all the flag was renamed in “clean start” and if set to 1, it means that the broker would discard any previous session (the client is asking for a “clean” start) otherwise it would keep session (the client is asking for not “cleaning” the current session).

Other than this change, the “session expiry interval” property was added (in the CONNECT packet) : after the client disconnects, the broker should remove session information for that client when this time is elapsed.

Delete please, if you can’t delivery on time …

Another really interesting property is the “publication expiry interval” which can be set into the PUBLISH message by the client. It’s something similar a TTL (Time to Live), as it already exists in the AMQP protocol, and it means : if this time has passed and the server, for any reason, can’t deliver the message to subscribers then it MUST delete this copy of the message.

In the IoT, it’s really common to use this feature for the “command and control” pattern, in order to avoid that offline devices start to execute “stale” commands when they come back online : if the command isn’t executed in a specified amount of time, it should be never executed.

The new enhanced authentication

Today, with the 3.1.1 specification, the binary value 1111 for the higher nibble of the first byte in the “fixed header” is forbidden/reserved. It’s changed in v5 because it represents the new AUTH packet.

Other than using the already available username/password built-in authentication, the AUTH packet provides a way to use a different authentication mechanism between client and server for including challenge/response style authentication; it’s something that the AMQP protocol supports with SASL mechanism for example.

Let others know that I’m dead but … not immediately

The “Last Will and Testament” (LWT) is a really cool feature which gives the possibilities to interested clients to know that another client is dead (without sending a “clean” disconnection packet). In the new specification, it’s possible to specify a “will delay” so that when the server detects a “not clean” disconnection from a client, it doesn’t send the “will message” immediately but after such a delay.

Keep alive timeout ? Now the server can decide !

With current 3.1.1 specification, the client sends a “keep alive timeout” value (in seconds) to the server : it represents the maximum amount of time between two packets sent by the client. If it expires, the server can consider the client dead (so sending the related “will message” for example). Up today, the client decided this value (disabling keep alive with a 0 value) but with new v5, the server can provide a “keep alive timeout” value in the CONNACK packet for the client : it means that the client MUST use this value instead the one it sent in the CONNECT packet.

Miscellaneous

Password … but no username

It’s now possible to send a password in the CONNECT packet without specifying the corresponding username. This allows to use the “password” field for credentials and not just for password.

Maximum packet size

Using the corresponding property, the client can now specify the maximum packet size it can accept.

Pipelining messages

The client can now starts to send other messages before receiving the CONNACK after sending the CONNECT packet. Of course, it means that if the server is sending the CONNACK with a negative reason code, it MUST NOT process any messages already sent by client.

Pipelining is one of the features already provided by AMQP but in a really more powerful way.

What identifier have you assigned to me ?

We know that the client-id provided by the client on connection is really useful for maintaining the correlation with session information. It’s also allowed to connect providing a “zero length” client-id but in this case the server will assign such identifier to the client. Today, the client doesn’t receive such information from the server while with the v5, the server provides the assigned client-id using the CONNACK packet.

I can’t handle higher QoS

Using the “maximum QoS” property in the CONNACK packet, the server can provide the maximum QoS level that it can handle for published messages. It means that if the client sends a packet with an higher QoS, the server will disconnect and the cool thing, with v5 specification, is that it will do that not just closing the TCP connection but providing a specific reason code (QoS not supported) in the DISCONNECT packet.

From topic name to … alias

A lot of people say that MQTT is lightweight but they don’t think that the topic name is specified in every message sent by the client (it’s different from AMQP where the client “attaches” on an address after the connection and can publish without specifying it anymore). The v5 specification adds the concept of “topic alias” through the corresponding property : it seems to be stolen by MQTT-SN protocol which provides a way to assign a single byte identifier to a topic name, so that in the subsequent PUBLISH packets, the client can avoid to specify the entire topic name but can use such identifier instead (it reduces the packet size).

Apache Kafka consumer groups … don’t use them in the “wrong” way !

8In this blog post I’d like to focus the attention on how the “automatic” and “manual” partitions assignment can interfere with each other even breaking things. I’d like to give an advice on using them in the right way avoiding to mix them in the same scenario or being aware of what you are doing.

The consumer group experience

In Apache Kafka, the consumer group concepts is a way for achieving two things :

  • having consumers as part of the same consumer group means providing the “competing consumers” pattern with whom the messages from topic partitions are spread across the members of the group; each consumer receive messages from one or more partitions (“automatically” assigned to it) and the same messages won’t be received by the other consumers (assigned to different partitions). In this way we can scale the number of the consumers up to the number of the partitions (having one consumer reading only one partition); in this case a new consumer joining the group will be in a idle state without being assigned to any partition.
  • having consumers as part of different consumer groups means providing the “publish/subscribe” pattern where the messages from topic partitions are sent to all the consumers across the different groups. It means that inside the same consumer group we’ll have the rules explained above but across different groups, the consumers will receive the same messages. It’s useful when the messages inside a topic are of interest for different applications which will process them in different ways; we want that the all the interested applications will receive all the same messages from the topic.

Another great advantage of consumers grouping is the re-balancing feature. When a consumer joins a group, if there are still enough partitions available (so we haven’t reached the limit of one consumer per partition), a re-balancing starts and the partitions will be reassigned to the current consumers plus the new one. In the same way, if a consumer leaves a group, the partitions will be reassigned to the remaining consumers.

What I have told so far it’s really true using the subscribe() method provided by the KafkaConsumer API. This method enforces you to assign the consumer to a consumer group, setting the “group.id” property, because it’s needed for re-balancing. In any case, it’s not the consumer to decide the partitions it wants to read for; in general the first consumer which joins the group doing the assignment while other consumers join the group.

How things can be “broken”

Other than using the subscribe() method, there is another way for a consumer for reading from topic partitions : it’s using the assign() method and in this case, the consumer is able to specify the topic partitions it wants to read for.

This type of approach can be useful when you know exactly where some specific messages will be written (the partition) and you want to read directly from there. Of course, you lose the re-balancing feature in this case and it is the first big difference with using the subscribe way.

Another difference is that with “manual” assignment you can avoid to specify a consumer group (so the “group.id” property) for the consumer : it will be just empty. In any case it’s better specifying it.

Most of the people use the subscribing way, leveraging “automatic” assignment and re-balancing feature; using both the way together can brake something … let’s see.

Imagine to have a single “test” topic with only two partitions (P0 and P1) and a consumer C1 which subscribe to the topic as part of the consumer group G1. This consumer will be assigned to both the partitions receiving messages from them. Now, let’s start a new consumer C2 which is configured to be part of the same consumer group G1 but … it uses the assign way for asking for the partitions P0 and P1 explicitly.

subscribe_assignNow we have broken something ! What ? Maybe you are asking …

Both the consumer C1 and C2 will receive messages from the topic, so from both partitions P0 and P1, but … they are part of the same consumer group G1 ! So we have “broken” what we said in the previous paragraph about “competing consumers” when they are part of the same consumer group. So you are experiencing a “publish/subscribe” pattern but with consumers within the same consumer group.

What about offsets commits ?

In general you should avoid a scenario like the one described before even because there is a real side effect on that. Starting from the version 0.8.2.0, the offsets committed by the consumers aren’t saved in Zookeeper but on a partitioned and replicated topic named “__consumer_offsets” which is hosted on the Kafka brokers in the cluster.

When a consumer commits some offsets (for different partitions), it really sends a message to the broker to the “__consumer_offsets” topic and such message has the following structure :

  • key = [group, topic, partition]
  • value = offset

Coming back to the previous scenario what does it mean ?

Having C1 and C2 as part of the same consumer group but able to receive from the same partitions (both P0 and P1) it could happen something like the following :

  • C1 commits offset X for partition P0 writing a message like this :
    • key = [G1, “test”, P0], value = X
  • C2 commits offset Y for partition P0 writing a message like this :
    • key = [G1, “test”, P0], value = Y

So the consumer C2 has overwritten the committed offset for the same partition P0 of the consumer C1 and maybe X was less than Y; if C1 crashes and restarts it will lose messages starting to read from Y (remember Y > X).

Something like that can’t happen with consumers which use only the subscribe way for being assigned to partitions, because as part of the same consumer group they’ll receive different partitions so the key for the offset commit message will be always different.

[Update 28/07/2017]

As a confirmation that mixing subscribe and assign isn’t a good thing to do, after a discussion with one of my colleagues, Henryk Konsek, it turned out that if you try to call both methods on the same consumer, the client library throws the following exception :

java.lang.IllegalStateException: Subscription to topics, partitions and pattern are mutually exclusive

Conclusion

The consumer groups mechanism in Apache Kafka works really well and leveraging on that for scaling consumers and having “automatic” partitions assignment with re-balancing is a great plus. There are cases where you would need to assign partitions “manually” but in that case pay attention on what could happen if you mix both solutions.

So … let’s consume from Apache Kafka but with judgment and the awareness of what we are doing.

A lot of fun with … AMQP, Spark, Kafka, EnMasse, MQTT, Vert.x & IoT

When I say to someone that I work for Red Hat they say me “Ah ! Are you working on Linux ?” … No, no, no and … no ! I’m not a Linux guy, I’m not a fan boy but I’m just a daily user 🙂

All people know that Red Hat is THE company which provides the best enterprise Linux distribution well known as Red Hat Enterprise Linux (RHEL) but Red Hat is not only Linux today. Its portfolio is huge : the cloud and containers business with the OpenShift effort, the microservices offer with Vert.x, Wildfly Swarm, Spring Boot, the IoT world with the involvement in the main Eclipse Foundation projects.

The objective of this blog is just showing briefly the projects I worked (or I’m working) on since last year when I was hired on March 1st. They are not “my” projects, they are projects I’m involved because the entire team is working on them … collaboration, you know 🙂

You could be surprised about that but … there is no Linux ! I’m on the messaging & IoT team, so you will see only projects about this stuff 🙂

AMQP – Apache Spark connector

This “little” component is strictly related to the “big” radanalytics.io project which takes the powerful of Apache Spark for analytics (batch, real-time, machine learning, …) running on OpenShift.

Because the messaging team works mainly on projects like ActiveMQ Artemis and the Qpid Dispatch Router, where the main protocol is AMQP 1.0, the idea was developing a connector for Spark Streaming in order to ingest data through this protocol so from queues/topics on a broker or through the router in a direct messaging fashion.

You can find the component here and even an IoT demo here which shows how it’s possible to ingest data through AMQP 1.0 using the EnMasse project (see below) and then executing a real time streaming analytics with Spark Streaming, all running on Kubernetes and OpenShift.

AMQP – Apache Kafka bridge

Apache Kafka is one of the best technologies used today for ingesting data (i.e. IoT related scenarios) with an high throughput. Even in this case, the idea was providing a way for having AMQP 1.0 clients and JMS clients pushing messages to Apache Kafka topics without knowing the related custom protocol.

In this way, if you have such clients because you are already using a broker technology but then you need some specific Kafka features (i.e. re-reading streams), you can just switch the messaging system (from the broker to Kafka) and using the bridge you don’t need to update or modify clients. I showed how this is possible at the Red Hat summit as well and the related demo is available here.

MQTT on EnMasse

EnMasse is an open source messaging platform, with focus on scalability and performance. It can run on your own infrastructure (on premise) or in the cloud, and simplifies the deployment of messaging infrastructure.

It’s based on other open source projects like ActiveMQ Artemis and Qpid Dispatch Router supporting the AMQP 1.0 protocol natively.

In order to provide support for the MQTT protocol, we designed how to take “MQTT over AMQP” so having MQTT features on the AMQP protocol. From the design we moved to develop two main components :

  • the MQTT gateway which handles connections with remote MQTT clients translating all messages from MQTT to AMQP and vice versa;
  • the MQTT LWT (Last and Will Testament) service which provides a way for notifying all clients connected to EnMasse that another client is suddenly died sending them its “will message”. The great thing about this service, is that it works with pure AMQP 1.0 clients so bringing the LWT feature on AMQP as well : for this reason the team is thinking to change its name just in AMQP LWT service.

EnMasse is great for IoT scenarios in order to handle a huge number of connections and ingesting a lot of data using AMQP and MQTT as protocols. I used it in all my IoT demos for showing how it’s possible to integrate it with streaming and analytics frameworks. It’s also the main choice as messaging infrastructure in the cloud for the Eclipse Hono project.

Vert.x and the IoT components

Vert.x is a great toolkit for developing reactive applications running on a JVM.

The reactive applications manifesto fits really well for IoT scenarios where responsiveness, resiliency, elasticity and the communication driven by messages are the pillars of all the IoT solutions.

Starting to work on the MQTT gateway for EnMasse using Vert.x for that, I decided to develop an MQTT server that was just able to handle communication with remote clients providing an API for interacting with them : this component was used for bridging MQTT to AMQP (in EnMasse) but can be used for any scenario where a sort of protocol translation or integration is needed (i.e. MQTT to Vert.x Event Bus, to Kafka, …). Pay attention, it’s not a full broker !

The other component was the Apache Kafka client, mainly developed by Julien Viet (lead on Vert.x) and then passed to me as maintainer for improving it and adding new features from the first release.

Finally, thanks to the Google Summer of Code, during the last 2 months I have been mentoring a student who is working on developing a Vert.x native MQTT client.

As you can see the Vert.x toolkit is really growing from an IoT perspective other then providing a lot of components useful for developing pure microservices based solutions.

Eclipse Hono

Eclipse Hono is a project under the big Eclipse IoT umbrealla in the Eclipse Foundation. It provides a service interfaces for connecting large numbers of IoT devices to a back end and interacting with them in a uniform way regardless of the device communication protocol.

It supports scalable and secure ingestion of large volumes of sensor data by means of its Telemetry API. The Command & Control API allows for sending commands (request messages) to devices and receive a reply to such a command from a device asynchronously in a reliable way.

This project is mainly developed by Red Hat and Bosch and I gave my support on designing all the API other then implementing the MQTT adapter even in this case using the Vert.x MQTT server component.

Because Eclipse Hono works on top of a messaging infrastructure for allowing messages exchange, the main choice was using ActiveMQ Artemis and the Qpid Dispatch Router even running them using Kubernetes and OpenShift with EnMasse.

Apache Kafka

Finally, I was involved to develop a POC named “barnabas” (a messenger character from a Frank Kafka novel :-)) in order to take Apache Kafka running on OpenShift.

Considering the stetaful nature of a project like Kafka, I started when Kubernetes didn’t offer the StatefulSets feature doing something similar by myself. Today, the available deploy is based on StatefulSets and it’s a work in progress on which I’ll continue to work for pushing the POC to the next level.

Apache Kafka is a really great project which has its own use cases in the messaging world; today it’s more powerful thanks to the new Streams API which allows to execute a real time streaming analytics using topics from your cluster and running simple applications. My next step is to move my EnMasse + Spark demo to an EnMasse + Kafka (and streaming) deployment. I’m also giving my support on the Apache Kafka code.

Conclusion

The variety and heterogeneity of all the above projects is giving me a lot of fun in my day by day work even collaborating with different people with different knowledge. I like learning new stuff and the great thing is that … things to learn are endless ! 🙂