Intel Joule debate and … #thesixthsense on #IoT

After Intel Developer Forum 2016 (IDF) with the new Joule IoT Dev Kit announcement, a debate started between some colleagues here in Red Hat on using this kind of platforms only for maker projects or in the enterprise world as well.

This debate became a post on the Red Hat Developers blog titled “Intel Joule Debate: A maker platform ready for widespread IoT use?”.

It was funny because before it was published, I started a #thesixthsense #IoT series on Twitter worried about how the embedded systems “could” be in a near future using high level tools by high level developers …

thesixthsense_iot

I think that the blog post worths a read and I hope you’ll enjoy it !

Open source nature and community for bringing .Net Micro Framework to a new life !

Yesterday, the biggest manufacturer of .Net Gadgeteer devices announced a bad news to the community: they won’t produce the boards based on it anymore.

To be precise, they won’t abandoned the entire .Net Micro Framework development but “only” all the boards known for the simplicity of developing prototypes avoiding soldering but using simple socket instead.

Of course, I’m speaking about the GHI Electronics.

I have some of their great boards used in a lot of demos and examples on .Net Micro Framework development for the Internet of Things during my sessions around Italy. Today, I can understand their decision: .Net Gadgeteer was created by Microsoft but today there isn’t so much effort on it and the community isn’t so huge for bringing the framework to the next level. In the last years, GHI Electronics was alone in order to support this framework and having it alive. From my point of view, it’s a pity because using .Net Gadgeteer boards, the ideas become reality in a very short time: from the advantage of using socket connection for the hardware to the application development with C# and .Net Micro Framework. The way to a final product is very long … but as starting point it was good.

To this bad news, we can add the silence by SecretLabs about another well-known board in the .Net Micro Framework world: the Netduino.

Netduino was my first love starting from its first version (without Ethernet) to the latest WiFi board. I played a lot with it, learning .Net Micro Framework development, even without the simplicity of hardware connections because I like “flying wires”. Even SecretLabs tried to do something like .Net Gadgeteer with Netduino Go without so much luck (they used a completely different “standard” for that).

The bad news and the silence let me start thinking about the future of the .Net Micro Framework.

Of course, to be precise, the framework life isn’t related to the future of the maker platforms provided by GHI Electronics and SecretLabs, because some companies are designing their own boards for running it.

It was born in the Microsoft research laboratories as the SPOT (Smart Personal Objects Technology) framework presented in the 2002. It became .Net Micro Framework and after some years Microsoft decided to provide it as open source without focusing so much on its development. Developers started to use it but only for hobbystic projects; only few companies decided to start developing real world products.

With the new IoT business, a couple of years ago Microsoft re-started to focus on it showing a lot of demos all around the world (at Build as well) based on .Net Micro Framework devices connected to Azure services (even before IoT Hub but using Service Bus and Event Hub). It seems that today this strong support is ended … Microsoft wants to provide better Cloud experience (the “I” in the IoT business) for all kind of devices without any difference on the running operating system or framework.

It seems to be another bad news for developers who believe in the .Net Micro Framework but fortunately … it’s always open source and the community can change its destiny.

As an open source developers (you know, I work for Red Hat … the company leader on open source) I believe in community projects : I started to write a lot of open sourced code for .Net Micro Framework a bunch of years ago and the community helped me to improve my libraries.

Today, Microsoft is providing few “human” resources to the .Net Micro Framework project hosted on GitHub in order to make the big changes on it and coordinate the community: I agree on that … now the community has the framework in its hands. At same time I know the Microsoft guys who are supporting it … they are great guys … trust me !

I know a lot of great guys who are working on .Net Micro Framework for their commercial products and for this reason I’m confident in the future. I know that the community around it isn’t so huge and it could be a problem for having a platform used as primary choice for embedded development. It’s also true that embedded world (yes ! embedded … not IoT !) is so much heterogeneous and each developer has its preferences: there is space for all of them, from ARM mbed platform to the .Net Micro Framework itself for example.

In the last months the interest about .Net Micro Framework was revamped by the new LILIUM project; to be precise it’s not the new version of the framework which has its own roadmap but a new way to develop in C# and UWP on embedded devices with the power of pre-compiled native code !

In order to define the right way for the community effort, two interesting discussions were opened on GitHub and all the main protagonists are arguing on the objectives and the roadmap: this is the essence of “to be a community”.

The first one is focused on the future of the .Net Micro Framework and its version 5 (today we are at 4.4) and the other one on the differences with LILIUM project; it’s worth for you to follow them both.

Let’s see … the IoT business opened a lot of scenarios … no doors are closed … can the open source nature and the community bring embedded C# development to a new life ?

M2Mqtt and GnatMQ are dead ? Long life to them !

In the last months a lot of people asked me if the M2Mqtt and GnatMQ projects are dead because there were no commits for a long time.

I can say … no ! They are not absolutely dead !

I was very busy in the last months, starting my new exciting Red Hat career in the messaging and IoT team, but I want to continue to support the projects as lead and main committer. Of course, I’ll be still busy with my new job but my commitment is to continue to improve the libraries, fixing bugs and adding new features. Just remember that I’ll do it in my spare time so I could not be so quick to reply your suggestions; of course your help with useful pull requests will be great !

As you now, today the M2Mqtt client library is under the official Paho GitHub repo and I have finally deleted the repo on my personal GitHub account.

The GnatMQ, based on M2Mqtt, will continue to live where it’s now here.

I’ll do my best to check the issues and the pull requests provided by the community, happy to know that my projects have a lot of followers and there are a lot of professional solutions using them out there !

As always I invite you to share with me your experience using my libraries in order to mention your solution in the “case studies” section of the official web site.

Thanks !

A routing IoT gateway to the Cloud

Let’s start with an on-premise solution …

Imagine that you have an embedded solution (or if you like it … an IoT solution) with a bunch of tiny devices which are connected to an on-premise server which receives telemetry data from them and is able to execute some elaboration in order to show information in real time on a dashboard and control the devices.

Imagine that your solution is based on the AMQP protocol and perhaps your on-premise server is running a messaging broker for gathering data from devices as messages through the local network.

Imagine that, due to your very constrained devices, the security in the network is guaranteed only at data level by encrypting the body of every single AMQP message. It’s possible that due to their complexity and need of more resources (CPU and memory) you can’t use sophisticated algorithms (i.e. DES, 3DES, AES, …) on your devices but only simple ones (i.e. TEA, ..).

Your solution is just working great in your environment.

… but now we want to move it to the Cloud

Imagine that for some reasons you need to change the on-premise nature of your solution and you want to connect the devices directly to the cloud with a very strict rule : nothing to change on the devices. At least you can change some configuration parameter (i.e. server ip, …) but not the way and the protocol they are using for communication.

The first simple solution could be moving your messaging broker from the on-premise server on a IaaS in the Cloud; just changing connection parameters on your devices and all continue to work as before.

The big problem now is that your data are sent through the public network and your security is based on a simple encryption algorithm applied only on the payload of the messages. For this reason, you start to think about using SSL/TLS in order to have security at connection level on top of TCP/IP, data encryption and server authentication.

Start to think about it but then … wait … I can’t use SSL/TLS on my tiny devices … they don’t have the needed resources in terms of CPU and memory … and now ?

Fog computing and IoT gateway : the solution ?

You know about “fog computing” (the new buzz word after IoT ?) and that you can solve your problem using an IoT gateway. Having this gateway could mean to have an intelligent piece of software which is able to gather data from the local network, process them in some way and then send them to the Cloud. The gateway could give you more features like filtering on data (sending only part of them), offline handling (if the Cloud isn’t reachable) and complex local processing but … wait … you don’t want it … you just want that data arrives to the Cloud in the same way as before (to the on-premise server) and for now you don’t need other additional great features.

Could we have a very simple IoT gateway with only the two following features we need :

  • SSL/TLS protocol support on behalf of the tiny devices;
  • traffic routing from devices to the Cloud in a transparent way;

The answer is … yes ! You have such solution and it’s provided by the Qpid Dispatch Router project from the ASF (Apache Software Foundation).

I already wrote about it in some previous articles [1] [3] so let me just show how you can use the router in a way that solve your “porting” problem.

The router just needs the right configuration

In order to show in a very simple way how to configure the router for our objective, we can use the Azure IoT Hub as Cloud platform for the IoT. As all the Azure messaging services like Service Bus and Event Hub, the IoT Hub needs an encrypted connection based on the SSL/TLS protocol … so it’s the problem we want to solve for our non SSL capable devices.

For the sake of simplicity we can run the router on a Raspberry Pi using the Raspbian distribution as OS; you can read about installing the Qpid Dispatch Router on Linux and on the Raspberry Pi in these articles [2] [4].

The main point is the configuration needed for the router in order to connect to an IoT Hub and routing the traffic from devices to it.

First of all we have to consider all the addresses that at AMQP level are used in order to send telemetry data to the hub, receive commands and reply with feedback. All these information are deeply explained here [5] [6].

The routing mechanism used in this configuration is the “link routing” [3] which means that the router creates a sort of “tunneling” between devices and the IoT Hub; it opens the TCP/IP connection with the hub, establishing it with SSL/TLS on top, and then opens the AMQP connection. All the SSL/TLS stuff happens between router and IoT Hub and the devices aren’t aware about it. You can see what happens through the router trace :

pi@raspberrypi:~ $ PN_TRACE_FRM=1 qdrouterd --conf ex06_iothub.conf
Sat Jul 23 11:56:17 2016 SERVER (info) Container Name: Router.A
Sat Jul 23 11:56:17 2016 ROUTER (info) Router started in Standalone mode
Sat Jul 23 11:56:17 2016 ROUTER_CORE (info) Router Core thread running. 0/Router.A
Sat Jul 23 11:56:17 2016 ROUTER_CORE (info) In-process subscription M/$management
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$management
Sat Jul 23 11:56:18 2016 AGENT (info) Activating management agent on $_management_internal
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$_management_internal
Sat Jul 23 11:56:18 2016 DISPLAYNAME (info) Activating DisplayNameService on $displayname
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$displayname
Sat Jul 23 11:56:18 2016 CONN_MGR (info) Configured Listener: 0.0.0.0:5672 proto=any role=normal
Listening on 0.0.0.0:5672
Sat Jul 23 11:56:18 2016 CONN_MGR (info) Configured Connector: ppatiernoiothub.azure-devices.net:5671 proto=any role=on-demand
Sat Jul 23 11:56:20 2016 POLICY (info) Policy configured maximumConnections: 0, policyFolder: '', access rules enabled: 'false'
Sat Jul 23 11:56:20 2016 SERVER (info) Operational, 4 Threads Running
Connected to ppatiernoiothub.azure-devices.net:5671
[0x19dc6c8]: -> SASL
[0x19dc6c8]:0 -> @sasl-init(65) [mechanism=:ANONYMOUS, initial-response=b"anonymous@raspberrypi"]
[0x19dc6c8]: -> AMQP
[0x19dc6c8]:0 -> @open(16) [container-id="Router.A", hostname="ppatiernoiothub.azure-devices.net", max-frame-size=65536, channel-max=32767, idle-time-out=60000, offered-capabilities=:"ANONYMOUS-RELAY", properties={:product="qpid-dispatch-router", :version="0.6.0"}]
[0x19dc6c8]: <- SASL
[0x19dc6c8]:0 <- @sasl-mechanisms(64) [sasl-server-mechanisms=@PN_SYMBOL[:EXTERNAL, :MSSBCBS, :ANONYMOUS, :PLAIN]]
[0x19dc6c8]:0 <- @sasl-outcome(68) 
[0x19dc6c8]: <- AMQP
[0x19dc6c8]:0 <- @open(16) [container-id="DeviceGateway_1766cd14067b4c4b8008b15ba75f1fd6", hostname="10.0.0.56", max-frame-size=65536, channel-max=8191, idle-time-out=240000]

At this point, the devices can connect locally to the router and when they asked for all the AMQP links related to the IoT Hub addresses, they will be tunneled by the router : the AMQP “attach” performatives are routed to the IoT Hub through the connection with the router. The communication then continues on this link in terms of message transfers directly between IoT Hub and devices but all encrypted until the router through the SSL/TLS protocol.router_iothub

The router configuration is something like that :

listener {
 addr: 0.0.0.0
 port: 5672
 authenticatePeer: no
}

ssl-profile {
 name: azure-ssl-profile
 cert-db: /opt/qdrouterd/Equifax_Secure_Certificate_Authority.pem
}

connector {
 name: IOTHUB
 addr: <iotHub>.azure-devices.net
 port: 5671
 role: on-demand
 sasl-mechanisms: ANONYMOUS
 ssl-profile: azure-ssl-profile
 idleTimeoutSeconds: 120
}

# sending CBS token
linkRoute {
 prefix: $cbs/
 connection: IOTHUB
 dir: in
}

# receiving the status of CBS token request
linkRoute {
 prefix: $cbs/
 connection: IOTHUB
 dir: out
}

# sending telemetry path and command replies from device to hub on : devices/<DEVICE_ID>/messages/events
# ATTENTION ! Here we need CBS Token
linkRoute {
 prefix: devices/
 connection: IOTHUB
 dir: in
}

# receiving command on device from hub on : devices/<DEVICE_ID>/messages/deviceBound
# ATTENTION ! Here we need CBS Token
linkRoute {
 prefix: devices/
 connection: IOTHUB
 dir: out
}

The main points in the configuration are :

  • a listener entity which defines that the router accept incoming AMQP connections on port 5672 (not encrypted);
  • the ssl-profile entity in order to configure the parameter for SSL/TLS connection to the IoT Hub and specifically the CA certificate to use for server authentication;
  • the connector entity which defines the way the router connects to the IoT Hub (address and port) using the above SSL profile;

After above parameters there is a bunch of linkRoute entities which define what are the addresses that should be link routed by the router from devices to the hub (using the specified connector).

You can find the complete configuration file here.

The Netduino Plus 2 use case

In order to develop an application very quickly on device side I decided to use my knowledge about .Net Micro Framework using a board that hasn’t the SSL/TLS support : the Netduino Plus 2 board.

The simple application is able to send a message to the IoT Hub and receive a new one replying with a feedback. All the code is available here.

In the following pictures you can see the message sent by the board and the command received (with the related feedback) through the Device Explorer tool.

01

02

Conclusion

Of course, the Qpid Dispatch Router project has a greater object than I showed here that could be providing connection to messaging services at scale thanks a more complex router network, with a path redundancy feature to reach a broker or a simple receiver.

In this article, I just showed a different way to use it in order to give more power to tiny devices which aren’t able to connect to AMQP based services due to their limitation (in this case the lack of SSL/TLS support).

If you consider the starting point, the configuration change could be avoided because the router could have same IP address and AMQP listening port as the previous on-premise server .

It means that only adding the router configured for the Cloud connection solves the problem !

[1] Routing AMQP : the Qpid Dispatch Router project

[2] Qpid Dispatch Router installation on your Linux machine

[3] Routing mechanisms for AMQP protocol

[4] My Raspberry Pi runs the Qpid Dispatch Router

[5] Connecting to the Azure IoT Hub using an AMQP stack

[6] Azure IoT Hub : commands and feedback using AMQP .Net Lite

My Raspberry Pi runs the Qpid Dispatch Router

This morning my “embedded” soul had an idea for supporting my “messaging” soul and spending some time in a productive way … trying to compile the Qpid Dispatch Router on a different machine with a different architecture : it’s C/C++ code … so it’s portable by definition (even if related to Linux for now) … it uses Python but today this language is available on a lot of different platforms … so it seems to be no problems.

Searching for some embedded stuff in my closet and discarding Cortex-Mx based boards (for obvious reasons regarding Linux lack on them) I got my Raspberry Pi … the first version (ARM11 based … not Cortex-Ax) :-)

embedded_stuff

I have the Pi2 as well (I have to by the third version) but I preferred to stop my research for starting immediately. I followed all the steps needed (explained in my previous article) using the Pi as a normal PC (connected via SSH) and after a while for compiling directly on the board, the router was up and running !

router_pi

The only tweak needed was to force cmake to use the Python 2.7 library with following command line :

cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DPYTHON_LIBRARY=/usr/lib/arm-linux-gnueabihf/libpython2.7.so -DPYTHON_INCLUDE_DIR=/usr/include/python2.7 -DPYTHON_EXECUTABLE=/usr/bin/python

because the 3.x version is installed on the Pi as well but we can’t use it for the router.

I know .. it’s not the right way to compile source code for embedded devices and cross-compilation from my PC is the better way to do that but I preferred it in order to go fast and avoid to setup a complete ARM toolchain on the laptop; of course be patient … building the Qpid Proton took me about one hour ! I can suggest you to have a good coffee …

Before starting the router, another simple tweak was needed in order to make persistent the value of the PYTHONPATH environment variable writing the following line to the .bashrc file :

export PYTHONPATH=/usr/lib/python2.7/site-packages

Now I have an idea … Pi with its Linux version is SSL/TLS capable but there are a lot of resources constrained devices which are able to “speak” AMQP but can’t support SSL/TLS connections. Why don’t use the Pi as a “shadow” IoT gateway and it’s security capabilities in order to bring above constrained devices to reach SSL/TLS and AMQP based cloud platforms even if they can’t “speak” SSL/TLS ?

Routing mechanisms for AMQP protocol

In the previous article, we installed the Qpid Dispatch Router and had a quick overview about the available tools inside the installation package for both router management and monitoring.

Now it’s time to start using the router from simple to complex configurations and network topologies with some examples which will involve AMQP senders, receivers and/or brokers. The broker part won’t be always necessary because the AMQP protocol is a “peer to peer” protocol and it works great connecting two clients directly without the “store and forward” mechanism provided by a broker in the middle. For more information about that you can read the first article of this series.

In this article, I’ll use the router with the default configuration showing how a sender and a receiver can connect to it and exchange messages through the router itself.

Routing mechanisms

First of all it’s interesting to say that the router supports two different types of routing mechanisms.

Message routing

When the router receives a message on a link, it uses the address specified in the target terminus when the sender attached the link itself to the route; if this address wasn’t specified, the destination address is get from the “To” property of the message. Based on such information, the router inspects its routing table to determine the route for delivering the message : it could be a link attached by a direct receiver to the router or another router inside the network that will be the next hop for reaching the destination. Of course, the message could be sent to different receivers all interested in the same address. The main point here is that the routing decision is made for each received message and there is always a communication between internal router nodes and external clients.

message_routing

As you can see in the above picture, a link is established between sender and router and between router and receiver. They are two completely distinct links that the router uses for messages exchange between sender and receiver through the routing mechanism on message basis.

For example, it means that there is a different flow control between router (with its internal receiver) and sender and between router (with its internal sender) and receiver : of course, it’s true that if the receiver grants few credits (i.e. 10) but the router (the internal receiver) grants more credits to the sender (i.e. 250 by default), it takes in account this difference. If for any reason the receiver closes the connection (after receiving 10 messages) and sender has already sent more than 10 messages (acknowledged by an “accepted” disposition), the router will reply with a “released” disposition for the next 40 messages because they can’t be delivered to the closed receiver.

Another interesting point is related to the message “settlement” : the router always propagates the delivery and its settlement along the network. On receiving a “pre-settled” message, its nature is propagated to the destination. The same is for “unsettled” messages : in that case, the router needs to track the incoming delivery and send the unsettled message to the destination; when it will receive the disposition (settlement) from the final receiver, it will reply in the same way to the original sender.

The last interesting aspect of message routing is the available routing patterns which define the paths followed by messages across the network :

  • closest : even if there are more receivers for the same address, the message is sent on the shortest path to reach the destination. It means that only one receiver will get the message;
  • balanced : when more receivers are attached to the same address, the messages sent to that address are spread across receivers. It means that each receiver will receive a different message at time in a sort of “competing consumers” way;
  • multicast : it’s something like a “publish/subscribe” pattern where all the receivers will receive the same message published on the address they are attached;

When the receivers for a specific address are all connected to the same router, we could think that “closest” and “balanced” have the same behavior because all the paths have same length and receivers are closed at same level to the router. It’s not so true because with “closest” the router uses a strict round-robin distribution across receivers while with “balanced” it takes into account the number of unsettled deliveries for each receiver and favors the “faster” of them (who settled messages faster than others).

To be more clear, suppose to have a sender S, two receivers R1 and R2 and a routers network with R1 connected to the same router as S and R2 connected to a different router (connected to the previous). We can say that R1 is “one level closed” to S and R2 is “two level closed” to S.

In the following scenario, with the “closest” distribution all the messages sent by S will be always delivered to R1.

closest_routing

“closest” message routing

Using the “balanced” distribution, the messages are spread across both receivers and there is nor relation with the path length.

balanced_routing

“balanced” message routing

Finally, with “multicast” distribution all messages are sent to all receivers.

multicast_routing

“multicast” message routing

Link routing

When the router receives an attach request, it’s propagated through the network in order to reach the destination node and establish the real link. When the sender starts to send messages to the router, it propagates that message through the established link to the destination without making any decision at message level. You can think of it as a sort of virtual connection or a tunnel between sender and receiver through a routers network.

link_routing

As you can see in the above picture, the link is established directly between the two peers and all performatives go through it.

From a flow control point of view, it’s directly handled between sender and receiver; any credits granted by the receiver arrives to the sender with a flow performative without any interference by the router in the middle. The link through the router is like a “tunnel” and it seems that the two peers are directly connected.

The same is true for disposition about settlement for “unsettled” messages that the sender receives directly from the receiver.

The concept of different routing patterns doesn’t make sense because in this case there is a direct link between sender and receiver so the router doesn’t make any decision on single message basis but it has only to propagate the frame along the link.

Let’s start … simple !

Now it’s time for a very simple example with a router started using the default configuration and only one sender and one receiver connected to it and attached to the “/my_address” address.

I used the “simple_recv” and “simple_send” C++ client examples for that and you can find them inside the Qpid Proton installation folder.

First of all let’s start the receiver specifying the address and the number of messages it wants to receive (it will grant link credits for that), i.e. 10 messages.

simple_recv_start

Using the qdstat management tool we can see that an endpoint for the “my_address” address is defined with “out” direction (from router to the receiver) with no messages delivered yet.

qdstat_01

After that let’s start the sender in order to send some auto generated messages, i.e. 5 messages.

simple_send_start

As you can see, the messages sent are all settled and confirmed. What’s happened at receiver side ?

simple_recv_messages

All messages sent by sender are now received and the simple application doesn’t close the connection because it’s waiting for the other 5 messages in relation to the 10 credits granted (of course it’s only an application behavior and not related to the router mechanisms).

Inspecting the router with the qdstat management tool we can see that on the output endpoint for the “my_address” address there are 5 delivered messages. What we can’t see is the endpoint on the same address with the opposite “in” direction (from sender to router) because after sending the messages, the sender closed the connection to the router and the endpoint is deleted. You can trust me … it was alive for all the time the sender was sending messages !

qdstat_02

As you can see we have directly connected sender and receiver without the need for a broker in the middle with its “store and forward” mechanism. In the above scenario, when the messages are settled and confirmed to the sender it means that they are really received by the receiver.

Conclusion

With this article I introduced the different mechanisms for messages routing that the Qpid Dispatch Router provides. For every scenarios we can choose the better way and what we need in order to distribute messages in a useful way for our distributed application. We saw a simple example on connecting sender and receiver through the router without the need for a broker in the middle. In the next articles, I’ll increase the complexity starting to use non default configuration files and exploring different way to connect routers with clients and brokers.

M2Mqtt : another moving … now under the official Eclipse Paho GitHub repo !

eclipse_github_repo

As you know they’re about 2 years that my M2Mqtt library is under the Eclipse Paho project as the official and supported C#, .Net (desktop, compact and micro) and WinRT client for MQTT protocol.

It’s source code was first released on CodePlex and then moved on GitHub inside my personal account in the last year but today it’s official part of the Eclipse Paho GitHub account.

It means that in the next few days I’m going to delete it as my personal GitHub repository and you will find all updates and fixes in its new “home”. Of course, you have to use this new repository in order to open new issues.

His brother, the GnatMQ broker, will stay on my personal GitHub account.

I’ll remain the principal committer and contributor but as open source project, the real power will come from the community !