.NetMF

GnatMQ : a more community driven future !

When I developed the M2Mqtt library, a lot of people asked me to develop a .Net based MQTT broker as well. It wasn’t my initial intention but at same time I thought that it could be a good idea trying to reuse part of the M2Mqtt client in order to do that : this is the way how GnatMQ was born.

From the beginning I said that it wasn’t a production ready MQTT broker with decent performance for its job but in the last years it turned out that a lot of developers started to use it for their IoT solutions.

Due to my last really busy year and pushed by this issue opened in the GnatMQ repo, this week I decided to create a GnatMQ organization and making this broker more open source then before. I added some developers who asked me to be member of such organization so that they can drive the GnatMQ future better than me : this members proved to contribute to the project in the past pushing PRs on it. Of course, I won’t give up on that, I will always feel like the project’s father and I’ll coordinate such community.

So … it means that GnatMQ isn’t dead … the community and the power of collaboration will move it to the next level, maybe having a really good fully .Net based MQTT broker !

 

Open source nature and community for bringing .Net Micro Framework to a new life !

Yesterday, the biggest manufacturer of .Net Gadgeteer devices announced a bad news to the community: they won’t produce the boards based on it anymore.

To be precise, they won’t abandoned the entire .Net Micro Framework development but “only” all the boards known for the simplicity of developing prototypes avoiding soldering but using simple socket instead.

Of course, I’m speaking about the GHI Electronics.

I have some of their great boards used in a lot of demos and examples on .Net Micro Framework development for the Internet of Things during my sessions around Italy. Today, I can understand their decision: .Net Gadgeteer was created by Microsoft but today there isn’t so much effort on it and the community isn’t so huge for bringing the framework to the next level. In the last years, GHI Electronics was alone in order to support this framework and having it alive. From my point of view, it’s a pity because using .Net Gadgeteer boards, the ideas become reality in a very short time: from the advantage of using socket connection for the hardware to the application development with C# and .Net Micro Framework. The way to a final product is very long … but as starting point it was good.

To this bad news, we can add the silence by SecretLabs about another well-known board in the .Net Micro Framework world: the Netduino.

Netduino was my first love starting from its first version (without Ethernet) to the latest WiFi board. I played a lot with it, learning .Net Micro Framework development, even without the simplicity of hardware connections because I like “flying wires”. Even SecretLabs tried to do something like .Net Gadgeteer with Netduino Go without so much luck (they used a completely different “standard” for that).

The bad news and the silence let me start thinking about the future of the .Net Micro Framework.

Of course, to be precise, the framework life isn’t related to the future of the maker platforms provided by GHI Electronics and SecretLabs, because some companies are designing their own boards for running it.

It was born in the Microsoft research laboratories as the SPOT (Smart Personal Objects Technology) framework presented in the 2002. It became .Net Micro Framework and after some years Microsoft decided to provide it as open source without focusing so much on its development. Developers started to use it but only for hobbystic projects; only few companies decided to start developing real world products.

With the new IoT business, a couple of years ago Microsoft re-started to focus on it showing a lot of demos all around the world (at Build as well) based on .Net Micro Framework devices connected to Azure services (even before IoT Hub but using Service Bus and Event Hub). It seems that today this strong support is ended … Microsoft wants to provide better Cloud experience (the “I” in the IoT business) for all kind of devices without any difference on the running operating system or framework.

It seems to be another bad news for developers who believe in the .Net Micro Framework but fortunately … it’s always open source and the community can change its destiny.

As an open source developers (you know, I work for Red Hat … the company leader on open source) I believe in community projects : I started to write a lot of open sourced code for .Net Micro Framework a bunch of years ago and the community helped me to improve my libraries.

Today, Microsoft is providing few “human” resources to the .Net Micro Framework project hosted on GitHub in order to make the big changes on it and coordinate the community: I agree on that … now the community has the framework in its hands. At same time I know the Microsoft guys who are supporting it … they are great guys … trust me !

I know a lot of great guys who are working on .Net Micro Framework for their commercial products and for this reason I’m confident in the future. I know that the community around it isn’t so huge and it could be a problem for having a platform used as primary choice for embedded development. It’s also true that embedded world (yes ! embedded … not IoT !) is so much heterogeneous and each developer has its preferences: there is space for all of them, from ARM mbed platform to the .Net Micro Framework itself for example.

In the last months the interest about .Net Micro Framework was revamped by the new LILIUM project; to be precise it’s not the new version of the framework which has its own roadmap but a new way to develop in C# and UWP on embedded devices with the power of pre-compiled native code !

In order to define the right way for the community effort, two interesting discussions were opened on GitHub and all the main protagonists are arguing on the objectives and the roadmap: this is the essence of “to be a community”.

The first one is focused on the future of the .Net Micro Framework and its version 5 (today we are at 4.4) and the other one on the differences with LILIUM project; it’s worth for you to follow them both.

Let’s see … the IoT business opened a lot of scenarios … no doors are closed … can the open source nature and the community bring embedded C# development to a new life ?

M2Mqtt and GnatMQ are dead ? Long life to them !

In the last months a lot of people asked me if the M2Mqtt and GnatMQ projects are dead because there were no commits for a long time.

I can say … no ! They are not absolutely dead !

I was very busy in the last months, starting my new exciting Red Hat career in the messaging and IoT team, but I want to continue to support the projects as lead and main committer. Of course, I’ll be still busy with my new job but my commitment is to continue to improve the libraries, fixing bugs and adding new features. Just remember that I’ll do it in my spare time so I could not be so quick to reply your suggestions; of course your help with useful pull requests will be great !

As you now, today the M2Mqtt client library is under the official Paho GitHub repo and I have finally deleted the repo on my personal GitHub account.

The GnatMQ, based on M2Mqtt, will continue to live where it’s now here.

I’ll do my best to check the issues and the pull requests provided by the community, happy to know that my projects have a lot of followers and there are a lot of professional solutions using them out there !

As always I invite you to share with me your experience using my libraries in order to mention your solution in the “case studies” section of the official web site.

Thanks !

A routing IoT gateway to the Cloud

Let’s start with an on-premise solution …

Imagine that you have an embedded solution (or if you like it … an IoT solution) with a bunch of tiny devices which are connected to an on-premise server which receives telemetry data from them and is able to execute some elaboration in order to show information in real time on a dashboard and control the devices.

Imagine that your solution is based on the AMQP protocol and perhaps your on-premise server is running a messaging broker for gathering data from devices as messages through the local network.

Imagine that, due to your very constrained devices, the security in the network is guaranteed only at data level by encrypting the body of every single AMQP message. It’s possible that due to their complexity and need of more resources (CPU and memory) you can’t use sophisticated algorithms (i.e. DES, 3DES, AES, …) on your devices but only simple ones (i.e. TEA, ..).

Your solution is just working great in your environment.

… but now we want to move it to the Cloud

Imagine that for some reasons you need to change the on-premise nature of your solution and you want to connect the devices directly to the cloud with a very strict rule : nothing to change on the devices. At least you can change some configuration parameter (i.e. server ip, …) but not the way and the protocol they are using for communication.

The first simple solution could be moving your messaging broker from the on-premise server on a IaaS in the Cloud; just changing connection parameters on your devices and all continue to work as before.

The big problem now is that your data are sent through the public network and your security is based on a simple encryption algorithm applied only on the payload of the messages. For this reason, you start to think about using SSL/TLS in order to have security at connection level on top of TCP/IP, data encryption and server authentication.

Start to think about it but then … wait … I can’t use SSL/TLS on my tiny devices … they don’t have the needed resources in terms of CPU and memory … and now ?

Fog computing and IoT gateway : the solution ?

You know about “fog computing” (the new buzz word after IoT ?) and that you can solve your problem using an IoT gateway. Having this gateway could mean to have an intelligent piece of software which is able to gather data from the local network, process them in some way and then send them to the Cloud. The gateway could give you more features like filtering on data (sending only part of them), offline handling (if the Cloud isn’t reachable) and complex local processing but … wait … you don’t want it … you just want that data arrives to the Cloud in the same way as before (to the on-premise server) and for now you don’t need other additional great features.

Could we have a very simple IoT gateway with only the two following features we need :

  • SSL/TLS protocol support on behalf of the tiny devices;
  • traffic routing from devices to the Cloud in a transparent way;

The answer is … yes ! You have such solution and it’s provided by the Qpid Dispatch Router project from the ASF (Apache Software Foundation).

I already wrote about it in some previous articles [1] [3] so let me just show how you can use the router in a way that solve your “porting” problem.

The router just needs the right configuration

In order to show in a very simple way how to configure the router for our objective, we can use the Azure IoT Hub as Cloud platform for the IoT. As all the Azure messaging services like Service Bus and Event Hub, the IoT Hub needs an encrypted connection based on the SSL/TLS protocol … so it’s the problem we want to solve for our non SSL capable devices.

For the sake of simplicity we can run the router on a Raspberry Pi using the Raspbian distribution as OS; you can read about installing the Qpid Dispatch Router on Linux and on the Raspberry Pi in these articles [2] [4].

The main point is the configuration needed for the router in order to connect to an IoT Hub and routing the traffic from devices to it.

First of all we have to consider all the addresses that at AMQP level are used in order to send telemetry data to the hub, receive commands and reply with feedback. All these information are deeply explained here [5] [6].

The routing mechanism used in this configuration is the “link routing” [3] which means that the router creates a sort of “tunneling” between devices and the IoT Hub; it opens the TCP/IP connection with the hub, establishing it with SSL/TLS on top, and then opens the AMQP connection. All the SSL/TLS stuff happens between router and IoT Hub and the devices aren’t aware about it. You can see what happens through the router trace :

pi@raspberrypi:~ $ PN_TRACE_FRM=1 qdrouterd --conf ex06_iothub.conf
Sat Jul 23 11:56:17 2016 SERVER (info) Container Name: Router.A
Sat Jul 23 11:56:17 2016 ROUTER (info) Router started in Standalone mode
Sat Jul 23 11:56:17 2016 ROUTER_CORE (info) Router Core thread running. 0/Router.A
Sat Jul 23 11:56:17 2016 ROUTER_CORE (info) In-process subscription M/$management
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$management
Sat Jul 23 11:56:18 2016 AGENT (info) Activating management agent on $_management_internal
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$_management_internal
Sat Jul 23 11:56:18 2016 DISPLAYNAME (info) Activating DisplayNameService on $displayname
Sat Jul 23 11:56:18 2016 ROUTER_CORE (info) In-process subscription L/$displayname
Sat Jul 23 11:56:18 2016 CONN_MGR (info) Configured Listener: 0.0.0.0:5672 proto=any role=normal
Listening on 0.0.0.0:5672
Sat Jul 23 11:56:18 2016 CONN_MGR (info) Configured Connector: ppatiernoiothub.azure-devices.net:5671 proto=any role=on-demand
Sat Jul 23 11:56:20 2016 POLICY (info) Policy configured maximumConnections: 0, policyFolder: '', access rules enabled: 'false'
Sat Jul 23 11:56:20 2016 SERVER (info) Operational, 4 Threads Running
Connected to ppatiernoiothub.azure-devices.net:5671
[0x19dc6c8]: -> SASL
[0x19dc6c8]:0 -> @sasl-init(65) [mechanism=:ANONYMOUS, initial-response=b"anonymous@raspberrypi"]
[0x19dc6c8]: -> AMQP
[0x19dc6c8]:0 -> @open(16) [container-id="Router.A", hostname="ppatiernoiothub.azure-devices.net", max-frame-size=65536, channel-max=32767, idle-time-out=60000, offered-capabilities=:"ANONYMOUS-RELAY", properties={:product="qpid-dispatch-router", :version="0.6.0"}]
[0x19dc6c8]: <- SASL
[0x19dc6c8]:0 <- @sasl-mechanisms(64) [sasl-server-mechanisms=@PN_SYMBOL[:EXTERNAL, :MSSBCBS, :ANONYMOUS, :PLAIN]]
[0x19dc6c8]:0 <- @sasl-outcome(68) 
[0x19dc6c8]: <- AMQP
[0x19dc6c8]:0 <- @open(16) [container-id="DeviceGateway_1766cd14067b4c4b8008b15ba75f1fd6", hostname="10.0.0.56", max-frame-size=65536, channel-max=8191, idle-time-out=240000]

At this point, the devices can connect locally to the router and when they asked for all the AMQP links related to the IoT Hub addresses, they will be tunneled by the router : the AMQP “attach” performatives are routed to the IoT Hub through the connection with the router. The communication then continues on this link in terms of message transfers directly between IoT Hub and devices but all encrypted until the router through the SSL/TLS protocol.router_iothub

The router configuration is something like that :

listener {
 addr: 0.0.0.0
 port: 5672
 authenticatePeer: no
}

ssl-profile {
 name: azure-ssl-profile
 cert-db: /opt/qdrouterd/Equifax_Secure_Certificate_Authority.pem
}

connector {
 name: IOTHUB
 addr: <iotHub>.azure-devices.net
 port: 5671
 role: on-demand
 sasl-mechanisms: ANONYMOUS
 ssl-profile: azure-ssl-profile
 idleTimeoutSeconds: 120
}

# sending CBS token
linkRoute {
 prefix: $cbs/
 connection: IOTHUB
 dir: in
}

# receiving the status of CBS token request
linkRoute {
 prefix: $cbs/
 connection: IOTHUB
 dir: out
}

# sending telemetry path and command replies from device to hub on : devices/<DEVICE_ID>/messages/events
# ATTENTION ! Here we need CBS Token
linkRoute {
 prefix: devices/
 connection: IOTHUB
 dir: in
}

# receiving command on device from hub on : devices/<DEVICE_ID>/messages/deviceBound
# ATTENTION ! Here we need CBS Token
linkRoute {
 prefix: devices/
 connection: IOTHUB
 dir: out
}

The main points in the configuration are :

  • a listener entity which defines that the router accept incoming AMQP connections on port 5672 (not encrypted);
  • the ssl-profile entity in order to configure the parameter for SSL/TLS connection to the IoT Hub and specifically the CA certificate to use for server authentication;
  • the connector entity which defines the way the router connects to the IoT Hub (address and port) using the above SSL profile;

After above parameters there is a bunch of linkRoute entities which define what are the addresses that should be link routed by the router from devices to the hub (using the specified connector).

You can find the complete configuration file here.

The Netduino Plus 2 use case

In order to develop an application very quickly on device side I decided to use my knowledge about .Net Micro Framework using a board that hasn’t the SSL/TLS support : the Netduino Plus 2 board.

The simple application is able to send a message to the IoT Hub and receive a new one replying with a feedback. All the code is available here.

In the following pictures you can see the message sent by the board and the command received (with the related feedback) through the Device Explorer tool.

01

02

Conclusion

Of course, the Qpid Dispatch Router project has a greater object than I showed here that could be providing connection to messaging services at scale thanks a more complex router network, with a path redundancy feature to reach a broker or a simple receiver.

In this article, I just showed a different way to use it in order to give more power to tiny devices which aren’t able to connect to AMQP based services due to their limitation (in this case the lack of SSL/TLS support).

If you consider the starting point, the configuration change could be avoided because the router could have same IP address and AMQP listening port as the previous on-premise server .

It means that only adding the router configured for the Cloud connection solves the problem !

[1] Routing AMQP : the Qpid Dispatch Router project

[2] Qpid Dispatch Router installation on your Linux machine

[3] Routing mechanisms for AMQP protocol

[4] My Raspberry Pi runs the Qpid Dispatch Router

[5] Connecting to the Azure IoT Hub using an AMQP stack

[6] Azure IoT Hub : commands and feedback using AMQP .Net Lite

Azure IoT Hub : get telemetry data using AMQP stack and Azure SB Lite

To complete the last two article series (data from device to cloud, command/feedback from cloud to device) on using AMQP protocol stack to interact with Azure IoT Hub, we need to get telemetry data from the devices.

When devices send data through their D2C endpoints, these data flow into the IoT Hub system and are made available to the service through its D2C endpoint that is Event Hubs compatible which means we can use any Event Hubs client to get data from it.

eventhubcompatible

On the new Azure portal we can see a lot of information about that in the related “Messaging” tab for our IoT Hub. The main information are :

  • Partitions : the number of partitions through which data from devices are ingested by the IoT Hub;
  • Event Hub compatible name : it’s the name of the event hub;
  • Event Hub compatible endpoint : it’s the complete path (with namespace) of the event hub;
  • Retention time : it’s the time the messages are retained inside the event hub;
  • Consumer groups : the available consumer groups for reading messages from event hub using related receivers (there is always the $Default consumer group);

To read from this event hub endpoint we can use any shared access policy that has the ServiceConnect permission. The portal provides us a default policy named “service” with its related shared access key. Using three of the above information we are able to build the connection string needed to connect to this auto-generated event hub :

Endpoint={Event Hub-compatible endpoint};SharedAccessKeyName={iot hub policy name};SharedAccessKey={iot hub policy key}

Of course, other than the above connection string we have to use the Event Hub compatible name.

Now … how the get data from event hub ? What’s the code we have to write ?

The simpler way to do that is to use the Event Processor Host provided by Microsoft as Nuget package that instantiates receivers for us on all available partitions and handle their leases providing us a checkpoint feature. The “only” big problem we have with this awesome software component is that it works only on .Net Framework so we can use it on PC based and Web based service applications.

UWP apps ? : AMQP and Azure SB Lite the solution

What can we do for UWP apps ? What we can do if we want to monitor telemetry data from an UWP app on Windows 10 without bridging data from a Web application ?

As in the previous articles the solution is to use a good C# implementation of AMQP protocol stack as AMQP .Net Lite but in this case, to avoid AMQP stuff, we can use the Azure SB Lite library (available on Nuget too) that wraps the protocol stack and exposes same official Service Bus SDK APIs to access to Event Hubs (other than queues and topics/subscriptions).

If you know the above APIs to interact with Event Hubs, the following code will be familiar to you :

static string ConnectionString = "Endpoint=[EVENT_HUB_COMPATIBLE_ENDPOINT];SharedAccessKeyName=[IOT_HUB_POLICY_NAME];SharedAccessKey=[IOT_HUB_POLICY_KEY]";
static string eventHubEntity = "[EVENT_HUB_COMPATIBLE_NAME]";
static string partitionId = "[PARTIION_ID]";
static DateTime startingDateTimeUtc;

static void Main(string[] args)
{
   ServiceBusConnectionStringBuilder builder = new ServiceBusConnectionStringBuilder(ConnectionString);
   builder.TransportType = TransportType.Amqp;

   MessagingFactory factory = MessagingFactory.CreateFromConnectionString(ConnectionString);

   EventHubClient client = factory.CreateEventHubClient(eventHubEntity);
   EventHubConsumerGroup group = client.GetDefaultConsumerGroup();

   startingDateTimeUtc = new DateTime(2015, 10, 31, 16, 00, 00);
            
   EventHubReceiver receiver = group.CreateReceiver(partitionId, startingDateTimeUtc);
            
   while (true)
   {
        EventData data = receiver.Receive();
        Debug.WriteLine("{0} {1} {2}", data.PartitionKey, data.EnqueuedTimeUtc.ToLocalTime(), Encoding.UTF8.GetString(data.GetBytes()));
   }

   receiver.Close();
   client.Close();
   factory.Close();
}

The above example is using Azure SB Lite and connecting to my current IoT Hub, it produces the following output :

iot_hub_event_hub_sample

Without having the Event Processor Host available on UWP apps, the bid deal is to create one or more receivers on all partitions by ourselves and handling the checkpoint on what is the position inside the stream we have already read. In that case, it’s useful to save the last reading date/time and use it as offset for starting a new read on the stream.

Conclusion

If you scaried to develop a monitoring UWP application because the related IoT Hub SDK doesn’t work on it … don’t worry you always have a solution that is based on using the underlying AMQP protocol stack. In that case, the solution is simpler thanks to the work I already done for the community with the Azure SB Lite. Of course, the choice to use and UWP application on a Windows 10 device instead of a .Net / Web application for monitoring your telemetry data is up to you and depends on the scenario, so it can make sense or not.

Azure IoT Hub : commands and feedback using AMQP .Net Lite

In the previous article, I described how it’s possibile to send telemetry data from a device and receive commands (sending feedback) from the cloud (to the device itself) using AMQP .Net Lite library connecting to the Azure IoT Hub. This approach is useful for some platforms that aren’t supported by the official SDKs like the .Net Micro Framework due to the Task and async/await programming model used in the C# implementation.

Covered the device side, it’s now time to see how we can use the same AMQP .Net Lite library on the service side. You could ask … “Why ?!! We have the great Microsoft.Azure.Devices Nuget package that provides the ServiceClient class to handle the connection, sending commands and receive feedback to/from devices !!”

You are right ! It’s true but … only if your service runs in a Web or a .Net Framework based application. If you try to install the above package in an UWP application you receive the following error !

nuget_uwp_error

As we can see, the official package uses some libraries that aren’t compatible with UAP like Microsoft.AspNet.WebApi.Core and Microsoft.Azure.Amqp. Tha latter sounds strange to me! It seems to be another AMQP stack implementation from Microsoft that can only run on .Net Framework. What are the differences with AMQP .Net Lite library? I’m sorry but I don’t know … the above stack isn’t open source and we can’t deep into it. However, there is a big and great difference that should be considered as a strength for AMQP .Net Lite : it works on all .Net platforms (micro, compact and full), on WinRT/UWP and Mono !

For this reason and using AMQP .Net Lite library we have a solution on developing an IoT Hub service inside a UWP application, a scenario that isn’t officially supported by Microsoft. As I love to say we can do that “knowing some AMQP protocol concepts and a bunch of IoT Hub node paths” !

IoT Hub service endpoints

Inside the IoT Hub architecture, the service has two endpoints to communicate with devices :

  • C2D (cloud to device) : the back end system can use this endpoint to send messages (for example commands) to the devices. This endpoint acts like a queue and each message has a TTL (Time To Live) so that it’s removed from the queue if the timeout expires (it’s useful to have commands executed in a short period of time and not executed too late when an offline device comes back online but the execution isn’t needed at that time because it could be harmful). The back end system can receive a confirmation message or delivery fault to understand if device has received command or not;
  • D2C (device to cloud) : it’s an Event Hubs compatible endpoint used by the back end system to retrieve messages from device (telemetry data) and feedback on command delivery (successful or not). “Event Hubs compatible” means that we can use an Event Hub client to receive messages from this endpoint (for example using an Event Processor Host implementation);

At AMQP level the endpoints are accessible from different entity paths; if you know Service Bus queues, topics/subscriptions and event hubs we can think them in the same way.

The entity path for sending command to devices is defined in the following way :

/messages/devicebound

while the entity path for receiving feedback (on commands sent) from devices is the following :

/messages/servicebound/feedback

As for the previous article, it means that after creating a connection and a session to our IoT Hub host we need to create two links to above entities (or nodes as defined in the AMQP spec). Using the programming model provided by AMQP .Net Lite library we have :

  • A SenderLink to the /messages/devicebound node;
  • A ReceiverLink to the /messages/servicebound/feedback node;

Authentication : sending the SAS token

The authentication mechanism is the same as device side. In this scenario, we need to send two SAS token on the two different AMQP nodes for sending command and receiving feedback.

The SAS token audience and resource URI for sending command are the same and defined in the following way :

string audience = Fx.Format("{0}/messages/devicebound", HOST);
string resourceUri = Fx.Format("{0}/messages/devicebound", HOST);

string sasToken = GetSharedAccessSignature(SHARED_ACCESS_KEY_NAME, SHARED_ACCESS_KEY, resourceUri, new TimeSpan(1, 0, 0));
bool cbs = PutCbsToken(connection, HOST, sasToken, audience);

For receiving feedback, they are the following :

string audience = Fx.Format("{0}/messages/servicebound/feedback", HOST);
string resourceUri = Fx.Format("{0}/messages/servicebound/feedback", HOST);

string sasToken = GetSharedAccessSignature(SHARED_ACCESS_KEY_NAME, SHARED_ACCESS_KEY, resourceUri, new TimeSpan(1, 0, 0));
bool cbs = PutCbsToken(connection, HOST, sasToken, audience);

Sending command

Using the SenderLink instance the device sends data calling the simple Send() method and passing it a Message class instance contains the data to send.

The sender link is created inside a new AMQP Session (using the related class of AMQP .Net Lite library) and the great news is that, thanks to the multiplexing feature of AMQP protocol, we can use the same session for both sender and receiver links all inside the same TCP connection.

The corresponding class in the official SDK is the ServiceClient class that provides the SendAsync() method. Regarding the original Message class (included into official SDK, not AMQP .Net Lite), it exposes the Ack property with following possible values :

  • none (default) : the service doesn’t want any feedback on command received by the device;
  • positive : the service receives a feedback message if the message was completed;
  • negative : the service receives a feedback message if the message expired (or max delivery count was reached) without being completed by the device;
  • full : the service receives both positive and negative feedbacks;

For more information you can refer to the previous article with a clear explanation of the message life cycle.

Using the AMQP .Net Lite library we don’t have an Ack property on the Message class but we need to use the application properties collection at AMQP level. The Ack property (at high level) is translated in an application property named “iothub-ack” (at AMQP level) which can have the above possible values. If we don’t set this application property, it means the same as “none” value so no feedback.

static private void SendCommand()
{
    string audience = Fx.Format("{0}/messages/devicebound", HOST);
    string resourceUri = Fx.Format("{0}/messages/devicebound", HOST);

    string sasToken = GetSharedAccessSignature(SHARED_ACCESS_KEY_NAME, SHARED_ACCESS_KEY, resourceUri, new TimeSpan(1, 0, 0));
    bool cbs = PutCbsToken(connection, HOST, sasToken, audience);

    if (cbs)
    {
         string to = Fx.Format("/devices/{0}/messages/devicebound", DEVICE_ID);
         string entity = "/messages/devicebound";

         SenderLink senderLink = new SenderLink(session, "sender-link", entity);

         var messageValue = Encoding.UTF8.GetBytes("i am a command.");
         Message message = new Message()
         {
              BodySection = new Data() { Binary = messageValue }
         };
         message.Properties = new Properties();
         message.Properties.To = to;
         message.Properties.MessageId = Guid.NewGuid().ToString();
         message.ApplicationProperties = new ApplicationProperties();
         message.ApplicationProperties["iothub-ack"] = "full";

         senderLink.Send(message);
         senderLink.Close();
    }
}

As we can see, the sending path “/messages/devicebound” hasn’t any information about the target device. To do that, the service need to set the To AMQP system property to the following value :

/devices/<DEVICE_ID>/messages/devicebound

where <DEVICE_ID> is the id assigned to the device when we create it inside the identity registry.

Finally, it’s importat to notice that the C2D endpoint queue can hold at most 50 messages.

Receiving feedback

Using the ReceiverLink instance the service can receive feedback from the device calling the Receive() method.

static private void ReceiveFeedback()
{
     string audience = Fx.Format("{0}/messages/servicebound/feedback", HOST);
     string resourceUri = Fx.Format("{0}/messages/servicebound/feedback", HOST);

     string sasToken = GetSharedAccessSignature(SHARED_ACCESS_KEY_NAME, SHARED_ACCESS_KEY, resourceUri, new TimeSpan(1, 0, 0));
     bool cbs = PutCbsToken(connection, HOST, sasToken, audience);

     if (cbs)
     {
          string entity = "/messages/servicebound/feedback";

          ReceiverLink receiveLink = new ReceiverLink(session, "receive-link", entity);

          Message received = receiveLink.Receive();
          if (received != null)
          {
               receiveLink.Accept(received);
               System.Diagnostics.Trace.WriteLine(Encoding.UTF8.GetString(received.GetBody<byte[]>()));
          }

          receiveLink.Close();
     }
}

The received message has a body in JSON format with an array of records (feedback from more different devices) each with following properties :

  • OriginalMessageId : it’s the MessageId of the original command (message) sent from the service to the device;
  • Description : description result that is related to the possible outcomes (success, message expired, maximum delivery count exceeded, message rejected);
  • DeviceGenerationId : device generation id related to the device that sent the feedback for a specific command;
  • DeviceId : device id related to the device that sent the feedback for a specific command;
  • EnqueuedTimeUtc : timestamp related to the outcome (it means when the feedback was enqueued);

For a single feedback, the JSON should be as following :

[{"originalMessageId":"5aac3169-af00-4536-acdb-cb9ea6b3980e","description":"Success","deviceGenerationId":"635794823643795743","deviceId":"<device_id>","enqueuedTimeUtc":"2015-10-29T07:59:00.9772497Z"}]

The full source code

As for all examples related to my blog posts, I update sample from previous article on GitHub. Now you can find a simple console application and a UWP application that are able to send command to a device and receive related feedback.

Conclusion

It’s clear that for a service running in a Web Application or .Net Framework based application, the best solution is to use the official Nuget package. With this article, I covered the NON officially supported possibility to use an UWP application to control devices through the IoT Hub thanks to the AMQP .Net Lite library and a bunch of AMQP paths.

If you consider the great portability of this AMQP stack implementation, you could control your devices using a .Net Compact Framework app (on Windows Embedded Compact 2013) or … another device based on .Net Micro Framework !

🙂

Connecting to the Azure IoT Hub using an AMQP stack

Accessing to the Azure IoT Hub service is so simple using all the available SDKs that Microsoft provides as open source on GitHub. We can laverage on more different languages for different platforms : from C# for .Net and UWP (Universal Windows Platform) applications to Java, from NodeJS (using JavaScript) to C. In this way, we are able to coverage different hardware from higher level devices like BeagleBoard, Raspberry Pi 2, Minnowboard Max to very low constrained devices like Kinetis K64F (Freescale) and CC3200 (Texas Instruments).

Of course, using directly HTTP and AMQP protocols, we can access to the IoT Hub from other platforms (not certified yet) and it could be the scenario of the .Net Micro Framework because up to now there isn’t an official SDK for it. The current C# client implementation uses Task and async/await programming model very heavily that isn’t supported (yet) by the “little” child in the .Net frameworks family. One choice to connect to the IoT Hub from a .Net Micro Framework board is to use directly the underlying AMQP protocol without the abstraction layer provided by an SDK. In this case you need to know some protocol concepts and a bunch of paths to the IoT Hub entities used for communication between devices and Cloud.

The purpose of this post is to show how to do this using the AMQP .Net Lite library that provides an AMQP protocol stack written in C# for a lot of different platforms like .Net Framework, .Net Compact Framework, .Net Micro Framework, WinRT (so UWP applications too), Mono, Windows Phone and so on; of course it’s available as open source on GitHub.

IoT Hub : connection and device endpoints

The IoT Hub is reachable using an address that has the following format

<IOT_HUB_NAME>.azure-devices.net

that we can retrieve from the Azure portal after creating the new IoT Hub instance service. As for all the services inside the Service Bus family (queues, topics/subscriptions and event hubs) the IoT Hub needs an SSL/TLS connection for data encryption and server authentication; it means that we have to connect to the host address to the default AMQPS (AMQP Secure) port that is the 5671.

We can create a new device inside the registry and get related credentials information using the Device Explorer application you can download here. After getting all needed information we can set them into the code.

private const string HOST = "IOT_HUB_NAME.azure-devices.net";
private const int PORT = 5671;
private const string DEVICE_ID = "DEVICE_ID";
private const string DEVICE_KEY = "DEVICE_KEY";

Using above information we can create an instance of the Address class and using it to establish the connection with the host thanks to the Connection class.

address = new Address(HOST, PORT, null, null);
connection = new Connection(address);

Inside the IoT Hub architecture, each device has two endpoints for accessing the Cloud :

  • D2C (device to cloud) : the device uses this endpoint to send messages to the cloud both as telemetry data and feedback for a received command (on the other endpoint, see below). It means that when we send a command to the device, it replies with a feedback at application level to confirm that the command is acquired and it’s going to be executed. Of course, it’s always true for a rejected command by the device;
  • C2D (cloud to device) : the device receives commands on this endpoint for executing the requested action. As already said, the device sends a confirmation (or rejection) of received command to the cloud using the other endpoint (D2C);

At AMQP level the endpoints are accessible from different entity paths; if you know Service Bus queues, topics/subscriptions and event hubs we can think them in the same way.

The entity path for sending data for telemetry purpose is defined in the following way :

/devices/<DEVICE_ID>/messages/events

where <DEVICE_ID> is the id assigned to the device when we create it inside the identity registry.

The entity path for receiving command from the Cloud is defined in the following way :

/devices/<DEVICE_ID>/messages/deviceBound

and as for the previous entity you need to provide the <DEVICE_ID> in the path.

It means that after creating a connection and a session to our IoT Hub host we need to create two links to above entities (or nodes as defined in the AMQP spec). Using the programming model provided by AMQP .Net Lite library we have :

  • A SenderLink to the /devices/<DEVICE_ID>/messages/events node;
  • A ReceiverLink to the /devices/<DEVICE_ID>/messages/deviceBound node;

Authentication : sending the SAS token

IoT Hub offers a per-device authentication through a SAS token that we can generate starting from device id and device key. After connection establishment we need to send such token to a specific CBS (Claim Based Security) endpoint to authorize the access to the specific entity.

As usual for Azure services, the token has the following format :

SharedAccessSignature sig={signature-string}&se={expiry}&skn={policyName}&sr={URL-encoded-resourceURI}

The big difference is that the skn field is absent in our case using device credentials .To get the SAS token I used the same code from my Azure SB Lite library because it’s processed almost in the same way.

string audience = Fx.Format("{0}/devices/{1}", HOST, DEVICE_ID);
string resourceUri = Fx.Format("{0}/devices/{1}", HOST, DEVICE_ID);
            
string sasToken = GetSharedAccessSignature(null, DEVICE_KEY, resourceUri, new TimeSpan(1, 0, 0));
bool cbs = PutCbsToken(connection, HOST, sasToken, audience);

The PutCbsToken creates a new session and a new link to connect to the specific $cbs node always using the same TCP connection. The content of the message is well defined by the AMQP CBS draft spec. After sending the token we are authorized to access IoT Hub from the device.

Just a note : I’m using the Fx class provided by AMQP .Net Lite library to have the Format method that doesn’t exist in the String class for the .Net Micro Framework.

Sending telemetry data

Using the SenderLink instance the device sends data calling the simple Send() method and passing it a Message class instance contains the data to send.

The sender link is created inside a new AMQP Session (using the related class of AMQP .Net Lite library) and the great news is that, thanks to the multiplexing feature of AMQP protocol, we can use the same session for both sender and receiver links all inside the same TCP connection.

static private void SendEvent()
{
    string entity = Fx.Format("/devices/{0}/messages/events", DEVICE_ID);

    SenderLink senderLink = new SenderLink(session, "sender-link", entity);

    var messageValue = Encoding.UTF8.GetBytes("i am a message.");
    Message message = new Message()
    {
        BodySection = new Data() { Binary = messageValue }
    };

    senderLink.Send(message);
    senderLink.Close();
}

Running the code, we can interact with the device using the Device Explorer application to receive the messages it sends.

dev_exp_iot_hub_1

Receiving command and send feedback

Using the ReceiverLink instance the device can receive command from the service in the Cloud calling the Receive() method. In addition to the sending commands features, the IoT Hub provides a feedback feature at application level for them; it means that the device is able to send a confirmation of received command to the service to accept or reject it. If the device is offline and doesn’t receive the command, the IoT Hub provides a TTL (Time To Live) you can set on every single message so that the command isn’t delivered to the device when it comes back online if the timeout is expired; this feature avoids to deliver a command that makes sense only if it’s executed on the device in a short time.

The device doesn’t need to send the feedback as a message on a specific AMQP node/entity but it’s handled by the IoT Hub when the ReceiverLink accepts or rejects the command. Using AMQP .Net Lite we can call the Accept() or Reject() methods on the ReceiverLink instance; at AMQP level it means that a “disposition” performative is sent to the IoT Hub with an outcome of “accepted” or “rejected”. Receiving this outcome the IoT Hub sends a feedback message to the D2C endpoint on the Cloud service side. With such outcomes the message goes into a completed state (positive feedback to the Cloud) or dead letter state (negative feedback).

static private void ReceiveCommands()
{
    string entity = Fx.Format("/devices/{0}/messages/deviceBound", DEVICE_ID);

    ReceiverLink receiveLink = new ReceiverLink(session, "receive-link", entity);

    Message received = receiveLink.Receive();
    if (received != null)
         receiveLink.Accept(received);

    receiveLink.Close();
}

Pay attention on the available Release() method in the library; in this case the outcome is “released” and the message returns into the command queue (enqueued state) ready to be re-delivered to the device if it calls the Receive() method again. If the device receives the messages more times and always calls the Release() method, the IoT Hub moves it into the dead letter state (removing it from the command queue) if the messages reaches the max delivery count; the same happens if the device doesn’t call neither Accept() nor Reject() methods and the TTL expires.

IoT Hub message lifecycle

IoT Hub message lifecycle

Executing the code and using Device Explorer to send the command we can see the feedback from the device too.

dev_exp_iot_hub_2

The full source code

The full source code I showed in the previous paragraphs is available on GitHub and it has projects for .Net Framework (so you can test very quickly it on your PC), generic .Net Micro Framework (for testing on your real device) and a third project for Netduino 3 WiFi as example of embedded device.

Of course, you can use any other board that support .Net Micro Framework and SSL/TLS protocol that is needed to connect to the IoT Hub. Other then Netduino 3 board, there are the FEZ Raptor and FEZ Spider from GHI Electronics (soon an example using them).

Conclusion

Knowing some AMQP concepts and some node paths you can find in the official IoT Hub Developer Guide, we are able to create an IoT Hub client using whatever AMQP stack implementation that in this example is AMQP .Net Lite; you can understand that it’s what the official IoT Hub client does creating an abstraction layer on top of it.

In this way we can add any other platform (not officially supported) like the .Net Micro Framework to the set of available devices for our Internet of Things solution based on IoT Hub.

A new Internet of Things era starts today … the IoT Hub is finally here !

September 29th … what a day for Microsoft and related Internet of Things technologies !

Azure IoT Suite and IoT Hub are finally here ! The IoT Hub is available in public preview and ready to help you to develop your IoT solutions in a very simple manner !

Where you can find all useful information about them ?

First of all the “Azure IoT Suite now available” blog post on official Microsoft web site focused on Internet of Things.

On Microsoft Azure documentation section you can find the dedicated “Azure IoT Developer Center” that will guide you to create a new IoT Hub on the new Azure management portal and connect one or more devices to it.

iot_dev_center

Starting with IoT Hub is very simple if you follow the documentation page here. A lot of examples, how-to guides for sending data from device to cloud and command from cloud to device, handling devices and so on.

Devices are a very interesting story !

There is a certified IoT program with all current hardware platforms and silicon vendors certified for accessing to IoT Hub. Of course, not only Raspberry Pi and MinnowBoard (we know them with Windows IoT Core) but BeagleBoard and Dragon Board 410C too. As embedded developer I love Freescale FRDM-K64F and Texas Instruments CC3200 LaunchPad too. It’s possible for you to become a partner and certify your hardware platform !

iot_hub_partners

To simplify your life, you can follow the main steps at this page for :

  • Select a device (Raspberry Pi 2, MinnowBoard Max, Freescale FRDM-K64F and so on)
  • Select a platform (Linux, Windows, mbed , TI RTOS). The list is filtered based on above selected device
  • Select a language (C, C#, JavaScript, Java). Pay attention … it’s not filtered …but remember you can’t use C# on Freescale FRDM-K64F 😉
  • Configure your IoT Hub with all steps on the new Azure management portal
  • Connect your device with all steps to do it

Great news … all Azure IoT SDKs are open source and available on GitHub !

A great partnership I like too much is with the ARM mbed platform as discussed on the ARM official blog. Currently only the Freescale FRDM-K64F board is supported but it’s a great starting point. On the mbed web site you can find the official Azure IoT account with a lot of source code related to the IoT Hub client implementation and related examples. Of course the client uses AMQP protocol to communicate with IoT Hub and for this reason the Qpid Proton C porting on mbed is available there. Just think … it can be useful to you to access all Service Bus entities (queues, topics/subscriptions, event hubs, …) from mbed enabled boards like the Freescale one. Remember that the Microsoft Band has a Kinetis MCU from Freescale (here the Sparkfun teardown) ! 😉

Of course .Net Micro Framework support on the boards from GHI or Secret Labs aren’t available “out of box” but using the great AMQP .Net Lite library you can use them with IoT Hub. Don’t worry … my open source developer soul is just thinking to a new project library for it 😉 It’s the same for industrial Windows Embedded Compact 2013 devices.

Regarding Azure IoT Suite and all related services for storing messages and predictive analysis there is a dedicated web site you can start to create a full IoT solution from predefined scenarios (with provisioning and monitoring of course) you can customize.

What if you have MQTT based devices or you want to develop new devices using this protocol ? How you can connect them to the Azure IoT Hub ? Don’t worry, Microsoft provides you the “Azure IoT Protocol Gateway” that is a framework for protocol adaptation that supports MQTT v3.1.1. You can deploy it in the Cloud as a worker role or in on-premises environments such as field gateways.

If all above material isn’t enough for you, there are some great videos on Channel9 from AzureCon :

In conclusion, today starts a new IoT era … after people hub and all the other hubs on your Windows Phone … you have the IoT Hub in the Cloud with all your devices !

Azure SB Lite … finally on Nuget !

azure_sb_lite_nuget

Finally my Azure SB Lite library is online on Nuget but … it was a long story 🙂

The last month, I decided to put this library on Nuget for all people that don’t like to compile source code from CodePlex and want to use it immediately. When I tried to upload the package, I received the “package id alredy used” error ! What ? Why ? Who ? Is there another library with same name ?

After few minutes I found that Sebastian Brandes (Microsoft Evangelist from Denmark) already uploaded it on Nuget in June (of course, he set me as “author”) and it was downloaded by 45 people. I contacted him who explained me that the library was useful to his project (thanks!) and that the right place for all other people was on Nuget (for much friendly usage). Right ! However, we decided that I should be the owner of the package so after about one month (Sebastian was on vacation) and Nuget support … today I re-uploaded the library !

I have to thank Sebastian for all his effort to help me to get the ownership of the package and for using it in his “Internet of Things” demo at Tech Ready conference (only for all Microsoft employees). He promised me to write a blog post and upload demo source code on GitHub.

Finally, I’d like to remember that there are some usage examples on GitHub at following link !

Internet of Things “wrap up” : eventi, slides, librerie e … demo con tanto codice !!

Nell’ultimo mese ho sviluppato molto ed ho avuto il piacere di essere speaker a tre eventi nazionali ed uno internazionale in California (ma da remoto).

Sono stato notevolmente impegnato nello scrivere codice e contenuti che ho deciso di riassumere qualora possano tornarvi utili.

DSC_0389 iotday

Con le slide ho cercato di descrivere al meglio l’Azure Service Bus focalizzandomi sull’Event Hubs e sul suo utilizzo nell’ambito della telemetria IoT :

A queste slide sono legate delle demo nelle quali ho utilizzato la mia ultima libreria Azure SB Lite, grazie alla quale è possibile accedere al Microsoft Azure Service Bus utilizzando le stesse API dell’SDK “originale” ma sulle piattaforme embedded come .Net Micro Framework e .Net Compact Framework oltre che su Mono (quindi Linux) e WinRT (quindi Windows 8.1 e Windows 10). Essa è basata sulla libreria AMQP .Net Lite di cui nasconde i concetti sul protocollo AMQP.

overall

Per quanto riguarda le demo :

  • BLE2Azure : concept di un IoT gateway da dispositive BLE verso Azure (Event Hubs) realizzato con il .Net Micro Framework e le board FEZ Spider e Netduino 3 Wi-Fi
  • Azure SB Lite Examples : esempi di utilizzo della libreria Azure SB Lite che mi hanno permesso (in quel di Venezia) di spiegare come funziona realmente l’accesso al Service Bus attraverso il protocollo AMQP e scoprire cosa ci sia al di sotto delle semplici APIs.

 Ovviamente non mi fermo qui nonostante il meritato riposo !

🙂