WPC 2015 Milan : Azure IoT Hub and IoT Suite

Organized by Overnet, in collaboration with Microsoft, WPC is the most important italian conference focused on Microsoft technologies. This year it will be covered in two full immersion days on December 1st and 2nd with 70 sessions in 8 tracks.

I’m honoured to be part of the speakers team this year as Microsoft MVP on Windows Embedded and IoT; on December 2nd, I’ll have a session about Microsoft Azure IoT Hub with an overview of the new Azure cloud gateway and the related Azure IoT Suite.

For sure, the conference will be great for contents and networking with all experts about Microsoft technologies. Don’t forget the “Ask The Expert” corner with a “bunch” of Microsoft MVPs ready to answer your questions.

All information and details about the registration and the conference on the official web site.

Let’s imagine Azure IoT Hub internal architecture

Today, thanks to the Microsoft Azure IoT Hub we can focus on developing our Internet of Things “end to end” solution at an application level perspective without concerning about the communication problems, the interconnection and messages exchange between the devices and  the service backend.

Before the advent of the IoT Hub we needed to setup all the communication channels to achieve the bidirectional paths from/to devices to/from Cloud. In that scenario, the best choice could be to use the Microsoft Azure Service Bus with Queues, Topics/Subscriptions and Event Hubs instances.

In the next paragraphs I’ll try to imagine (at very high level) what the IoT Hub service provides internally for us and how it sets up all the mentioned channels; we could mimic the related architecture using a bunch of Service Bus entities. During my explanation, I’ll use terms like “may” and “should” because I don’t know how it works for real, I can only imagine it and thinking as I need to implement it from scratch.

I consider this post as a conclusion of my previous “trilogy” on how to connect to the Azure IoT Hub using an AMQP stack, that is useful to understand how it works internally; these articles covered how to connect from a device perspective, how to handle command and feedback and finally how to get telemetry from devices.

The telemetry path

The first “simple” path we need to setup for an IoT solution is the telemetry one related to messages flow from devices to the Cloud without any response or feeback in the opposite direction. To support the ingestion of million events/second the IoT Hub “should” use an Event Hub like mechanism and it “may” be true because the D2C endpoint (at the service side) is defined as “Event Hub compatible” and we can read from it using a “pure” Event Hub client (like “low level” Event Hub Receiver or “high level” Event Processor Host).

As explained in this blog post, at AMQP level the devices sends data using a link connected to the following node as D2C endpoint (at device side) :

/devices/<DEVICE_ID>/messages/events

and the same node is exposed as “Event Hub compatible” at D2C endpoint at Cloud side (as already mentioned at this blog post). The related information ar available on the Azure portal to build the endpoint connection string.

It should be clear that the telemetry path is achieved using an Event Hubs like channel.

The command path …

For handling commands from service to device, we need a channel for sending them and another one to receive feedback about their delivery (accepted, rejected, expired, …).

As explained in this blog post, the command path is achieved using a link to the following node at service side :

/messages/devicebound

and this node at device side :

/devices/<DEVICE_ID>/messages/deviceBound

The command path “should” be a queue on both sides (devices and service) with a related TTL (time to leave) and dead letter queue for expired or rejected messages by devices.

As we can see, the sending path “/messages/devicebound” hasn’t any information about the target device. To do that, the service need to set the To AMQP system property to the following value. An internal mechanism “shoud” route the command to the right queue related to the destination device analyzing the message and reading the above To property.

It means that the internals “shoud” provides a queue on service side for sending commands and a queue for each device for receiving them.

… and feedback path

When the device accepts or rejects the message received on its C2D endpoint, the IoT Hub internals generates a feedback that is sent to another possible queue mapped on the following path :

/messages/servicebound/feedback

In this case, the device information related to the feedback are inside the body of the message itself in JSON format as described in the following post.

In this case, the feedback path “should” be implemented with a queue on service side.

Conclusion

As you can see, the IoT Hub “should” provision a bunch of “Service Bus – like” entities for us inside a unique namespace related to the IoT Hub itself. Before this new services, we needed to setup all the event hubs and queues instance by ourselves … today IoT Hub provides the entire architecture.

As I said … it’s only my imagination but … a possible high level solution to implement IoT Hub internally. It’s only an analogy game with a “home made” solution as you can see from the following Twitter conversation about this post by Clemens Vasters and Olivier Bloch from Service Bus and IoT Hub teams in Microsoft.

Azure IoT Hub and IoT Suite : my chat with DotNetPodcast team

For all italians people (or all my foreign friends who can understand italian ;-)) I’d like to announce another podcast about the Internet of Things on DotNetPodcast.

This is my third podcast on this stuff and I want to thank the DotNetPodcast team (Roberto Albano, Antonio Giglio, Massimo Bonanni) who invited me another time. It’s a pleasure for me.

This time I speak about the new Microsoft Azure managed service for the IoT world : the IoT Hub.

Why the need for the IoT Hub, what are its main features, connectivity and supported protocols, security, SDKs and certified hardware and how it fits well in an end to end IoT solution built using the Azure IoT Suite. Finally a brief comparison with the competitor AWS IoT from Amazon. These are the main points of my chat that you can find here.

I hope you’ll enjoy it !

Azure IoT Hub : get telemetry data using AMQP stack and Azure SB Lite

To complete the last two article series (data from device to cloud, command/feedback from cloud to device) on using AMQP protocol stack to interact with Azure IoT Hub, we need to get telemetry data from the devices.

When devices send data through their D2C endpoints, these data flow into the IoT Hub system and are made available to the service through its D2C endpoint that is Event Hubs compatible which means we can use any Event Hubs client to get data from it.

On the new Azure portal we can see a lot of information about that in the related “Messaging” tab for our IoT Hub. The main information are :

• Partitions : the number of partitions through which data from devices are ingested by the IoT Hub;
• Event Hub compatible name : it’s the name of the event hub;
• Event Hub compatible endpoint : it’s the complete path (with namespace) of the event hub;
• Retention time : it’s the time the messages are retained inside the event hub;
• Consumer groups : the available consumer groups for reading messages from event hub using related receivers (there is always the $Default consumer group);

To read from this event hub endpoint we can use any shared access policy that has the ServiceConnect permission. The portal provides us a default policy named “service” with its related shared access key. Using three of the above information we are able to build the connection string needed to connect to this auto-generated event hub :

Endpoint={Event Hub-compatible endpoint};SharedAccessKeyName={iot hub policy name};SharedAccessKey={iot hub policy key}

Of course, other than the above connection string we have to use the Event Hub compatible name.

Now … how the get data from event hub ? What’s the code we have to write ?

The simpler way to do that is to use the Event Processor Host provided by Microsoft as Nuget package that instantiates receivers for us on all available partitions and handle their leases providing us a checkpoint feature. The “only” big problem we have with this awesome software component is that it works only on .Net Framework so we can use it on PC based and Web based service applications.

UWP apps ? : AMQP and Azure SB Lite the solution

What can we do for UWP apps ? What we can do if we want to monitor telemetry data from an UWP app on Windows 10 without bridging data from a Web application ?

As in the previous articles the solution is to use a good C# implementation of AMQP protocol stack as AMQP .Net Lite but in this case, to avoid AMQP stuff, we can use the Azure SB Lite library (available on Nuget too) that wraps the protocol stack and exposes same official Service Bus SDK APIs to access to Event Hubs (other than queues and topics/subscriptions).

If you know the above APIs to interact with Event Hubs, the following code will be familiar to you :

static string ConnectionString = "Endpoint=[EVENT_HUB_COMPATIBLE_ENDPOINT];SharedAccessKeyName=[IOT_HUB_POLICY_NAME];SharedAccessKey=[IOT_HUB_POLICY_KEY]";
static string eventHubEntity = "[EVENT_HUB_COMPATIBLE_NAME]";
static string partitionId = "[PARTIION_ID]";
static DateTime startingDateTimeUtc;

static void Main(string[] args)
{
   ServiceBusConnectionStringBuilder builder = new ServiceBusConnectionStringBuilder(ConnectionString);
   builder.TransportType = TransportType.Amqp;

   MessagingFactory factory = MessagingFactory.CreateFromConnectionString(ConnectionString);

   EventHubClient client = factory.CreateEventHubClient(eventHubEntity);
   EventHubConsumerGroup group = client.GetDefaultConsumerGroup();

   startingDateTimeUtc = new DateTime(2015, 10, 31, 16, 00, 00);
            
   EventHubReceiver receiver = group.CreateReceiver(partitionId, startingDateTimeUtc);
            
   while (true)
   {
        EventData data = receiver.Receive();
        Debug.WriteLine("{0} {1} {2}", data.PartitionKey, data.EnqueuedTimeUtc.ToLocalTime(), Encoding.UTF8.GetString(data.GetBytes()));
   }

   receiver.Close();
   client.Close();
   factory.Close();
}

The above example is using Azure SB Lite and connecting to my current IoT Hub, it produces the following output :

Without having the Event Processor Host available on UWP apps, the bid deal is to create one or more receivers on all partitions by ourselves and handling the checkpoint on what is the position inside the stream we have already read. In that case, it’s useful to save the last reading date/time and use it as offset for starting a new read on the stream.

Conclusion

If you scaried to develop a monitoring UWP application because the related IoT Hub SDK doesn’t work on it … don’t worry you always have a solution that is based on using the underlying AMQP protocol stack. In that case, the solution is simpler thanks to the work I already done for the community with the Azure SB Lite. Of course, the choice to use and UWP application on a Windows 10 device instead of a .Net / Web application for monitoring your telemetry data is up to you and depends on the scenario, so it can make sense or not.