M2M

GnatMQ and SSL/TLS support : make it up and running !

After the last blog post on the new GnatMQ release with SSL/TLS support, it’s now time to show how to use this new feature.

Asymmetric encryption and SSL/TLS : a brief introduction

Speaking about SSL/TLS security means to speak a lot about asymmetric encryption (even if symmetric encryption is used as well) and X.509 certificates. Regarding asymmetric encryption we know that it’s based on public and private keys compared to symmetric encryption that uses only one shared key. Simplifying how asymmetric encryption works we can say that the message sender needs the public key of the receiver in order to use it for encrypting data. The receiver is the only one to have its private key so that it’s the only one that is able to decrypt a message that was encrypted with its own related public key. It means that public key can be distributed publically but private key must be kept secret. Asymmetric encryption can be used not only to encrypt data but for signing it too. In this case, the sender processes an hash on the message to send using its private key; the receiver can use the corresponding sender public key to verify the signature (the hash) and in this way it can be sure that the message was received from the right sender.

Of course, it’s only a simple and brief introduction on how asymmetric encryption works and I can suggest you to deep into it finding a lot of content online.

How is asymmetric encryption related to SSL/TLS protocol ?

SSL/TLS is a protocol on top of TCP/IP that is used for server authentication (to know that we are speaking with the server we want really to speak), client authentication (check on server side that the client can connect to the server itself) and content encryption (we can guarantee confidentiality on transmitted data). SSL/TLS is based on asymmetric encryption in the following way :

  • the server has a private key for encrypting messages during the first connection phase named SSL/TLS handshake;
  • the server has a related public key distributed publically to all clients want to connect to it;

I said that the server private key is used only during SSL/TLS handshake and not for encrypting all messages exchanged during connection. It’s true because during this phase, client and server exchange some information to generate a symmetric key that they will use during communication to encrypt all exchanged data. Why this step from asymmetric to symmetric encryption ?

Asymmetric encryption is more complex than symmetric and it needs more CPU and memory footprint for executing even if it’s more secure than symmetric. This one is less complex and doesn’t need huge resources for processing. This is the reason why for generating a symmetric key, the client and server use asymmetric encryption with more load during the handshake but they use the generated symmetric key for encrypting all traffic to be much faster during communication.

Now, the big question is : how the client can be sure it’s talking with the server it’d like to talk ? Here the X.509 certificates standard and the Certification Authority concepts come in and of course the second possible usage of asymmetric encryption : the signature.

The server public key must be signed by a trusted third party authority so that the client can trust this public key and check that it’s talking with the right server. A Certification Authority (CA) is a trusted third party that distributes X.509 certificates : you can think about a certificate as a server public key with a signature from CA (with more other information). It means that we can buy a server certificate from a CA directly or we can ask the CA to sign our already generated public key. Of course the CA uses its own private key to process the hash/signature for that public key to obtain a valid X.509 certificate.

How the client is able to check the server certificate validity ? Of course, the client needs the CA certificate (its public key) and can use it to verify the signature on the received certificate from the server.

Now you could ask … who signs the CA certificate ? In this case the CA is considered as a “root” so no other trusted third party over it; for this reason the CA certificate is self signed and it means that it’s signed by the CA itself … you need to trust it ! However you need to know that there are CA certificates not self signed but signed by other CAs; in this way there is the possibility to have CAs who can distribute X.509 certificates to companies but you can’t consider them as “root” because their certificates are signed by other “root” CAs.

After this higher level introduction to SSL/TLS (you need to deep into it for understanding it better), it’s time to “play” with certificates and prepare our GnatMQ broker with the new security feature.

Self-signed CA certificate creation

First of all we need a CA certificate to generate and sign the server certificate. We could buy the server certificate from a trusted third party Certificate Authority such as GoDaddy, Symantec (Verisign), Comodo or GlobalSign but only for testing, we can first generate a self-signed CA certificate and then using it to sign our server certificate. With our CA certificate we are able to impersonate a trusted Certification Authority so that we can sign any other certificate we’d like to use for our servers.

Remember that the certificates story starts from generating a public and private key for asymmetric cryptography. A server public key is signed with a CA private key to obtain a so called “certificate”. In this way the server public key can be distributed publically in a certificate fashion. In the case of a self-signed CA certificate, the CA private key is used to sign the same CA public key to obtain the self-signed CA certificate itself.

To execute all available operations on X.509 certificates (create, sign, convert and so on) there are two main available tools : OpenSSL or Makecert.
During this article we’ll use Makecert that we receive with our Visual Studio 2015 installation (it’s available with previous versions as well).
Of course, the certificate generation operation will generate both private key and public key; the last one will be signed to become a standard X.509 certificate to distribute publically.
Let’s start with the “Developer Command Prompt for VS2015” and check that the Makecert usage is the following :
makecert [options] outputCertificateFile
with a lot of “options” we can specify and the output certificate file.
Let’s execute following command :
makecert -n "CN=MyCARoot" -r -a sha512 -len 4096 -cy authority -sv MyCARoot.pvk MyCARoot.cer
01_gnatmq_ssl
where options have following meaning :
  • -n : specifies the so called Common Name and it’s the subject certificate name that conform to the X.500 standard
  • -r : ask for creation of a self-signed certificate
  • -a : specifies what algorithm must be used to sign (in this case self-signed) the certificate
  • -len : specifies the generated private key length, in bits
  • -cy : specifies the certificate type. Valid values are “end” for end-entity and “authority” for Certification Authority
  • -sv : specifies the name of the generated subject’s private key file

It’s important to notify the the -n parameter can receive more other options like Country Name (C), Organization Name (O), Locality Name (L) and so on; in this case a more complex value could be : “CN=MyCARoot,C=Italy,O=MyCACompany,L=Naples”.

There are a lot of other options you can specify like the expiration date and so on. For more information, you can see the official documentation for Makecert tool.

After executing this command you are prompted to insert the “passphrase” (a password) to protect the private key.

02_gnatmq_ssl
The first time you need to create the password for the private key and the second time you are prompted to insert the previous one to access the private key and (self)signing the certificate.
03_gnatmq_ssl
Now we have the MyCARoot.cer file and the MyCARoot.pvk file. The former is the self-signed CA certificate in the DER (binary) format and the latter is the private key used to signed the generated public key to obtain the above certificate and that will be used to sign any other server certificate.

Create the server certificate

Next step is to create the server certificate and sign it with the above CA certificate (using the related private key of course).
makecert -n "CN=DESKTOP-862CII2" -iv MyCARoot.pvk -ic MyCARoot.cer -pe -a sha512 -len 4096 -sky exchange -sv GnatMQ.pvk GnatMQ.cer
where options have following meaning :
  • -n : specifies the Common Name of the server. It must match the domain name or the PC name (like for this example)
  • -iv : specifies the issuer’s .pvk private key file (so the CA private key file)
  • -ic : specifies the issuer’s certificate file (so the self-signed CA certificate)
  • -pe : marks the generated private key as exportable (we’ll see what it means)
  • -a : specifies what algorithm must be used to sign the certificate
  • -len : specifies the generated private key length, in bits
  • -sky : specifies the subject’s key type.The “exchange” value means the key is used for key encryption and key exchange (what happes during SSL/TLS handshake)
  • -sv : specifies the name of the generated subject’s private key file

04_gnatmq_ssl

As for the CA certificate generation you are prompted to insert the passphrase to protect the private server key. In this case, you are prompted to insert the passphrase for the CA private key too (the Issuer) because it is necessary to access the CA private key for signing the server public key and generate the certificate.

05_gnatmq_ssl

In order to use the server certificate inside the broker we need to export it in the PFX format (predecessor of current PKCS#12). The PKCS#12 evolved from the personal information exchange (PFX) standard and is used to exchange public and private objects in a single file. It means that the single PFX file contains both the public and private key of the server. In this way, it has all needed data to support an SSL/TLS connection for deploying its certificate and encrypt data using the private key. This is the reason way we used the “pe” option in the Makecert command to mark the private key as exportable into the PFX file.

To do this we can use the pvk2pfx tool in the following way :

pvk2pfx -pvk GnatMQ.pvk -spc GnatMQ.cer -pfx GnatMQ.pfx -po <password>

where options have following meaning :

  • -pvk : specifies the name of a .pvk file (broker private key)
  • -spc : specifies the name and extension of the Software Publisher Certificate (SPC) file that contains the certificate (broker certificate)
  • -pfx : specifies the name of the output .pfx file
  • -po : password for the .pfx file

06_gnatmq_ssl

Setup GnatMQ security

Now we are able to use the server certificate to create a GnatMQ instance with SSL/TLS support. You can get the latest version as Nuget package and reference it inside your project. You can use the MqttBroker class instance inside a simple console application or a worker role on Azure; it’s up to you how to use it. For a simple example, the console application is the right choice.

First of all you need to add the generated GnatMQ.pfx file as resource inside your project to make it available in your source code.

07_gnatmq_ssl

The source code is so simple …

class Program
{
    static void Main(string[] args)
    {
        X509Certificate2 serverCert = new X509Certificate2(Resource1.GnatMQ, "<pwd>");

        // create and start broker
        MqttBroker broker = new MqttBroker(serverCert, MqttSslProtocols.TLSv1_0);
        broker.Start();

        Console.ReadLine();

        broker.Stop();
    }
}

In order to use the PFX file you can instantiate the X509Certificate2 class providing it the raw bytes and the password (you specified during PFX file creation). The MqttBroker class has a simple constructor for SSL/TLS support with only two parameters :

  • the server certificate;
  • the SSL/TLS protocol version;

In this way the broker is up and running and can accept only SSL/TLS connections !

Another way to instantiate the class is to use the constructor with other parameters like the RemoteCertificateValidationCallback and LocalCertificateSelectionCallback that you can use to customize the certificate validation and selection process.

Remember that any client you use to connect the broker it needs the self-signed CA certificate and your connection will be protected and encrypted after the server authentication.

Last one thing to know is that the SSL/TLS feature is supported by GnatMQ only for the .Net Framework platform and not for the WinRT (Windows 8.1 and Windows 10) due to the sandboxed execution environment that doesn’t accept incoming SSL/TLS connection (only as client).

Windows IoT Core and M2Mqtt … a simple marriage !

“Hello, Windows IoT Core” as Steve Texeira wrote few days ago !

The last stable release of the new IoT (embedded ?) OS for makers (and professionals ?) is out but of course I couldn’t wait the last days to start using it so I already played with it in a lot of demos and sessions code.

Who follows me knows that other than interacting with real world using sensors from Raspberry Pi 2, my great love is share these data and send them to the Cloud ! I already spoke and coded about using Pi 2 to send data to the awesome Event Hubs service (in the Microsoft Azure Service Bus family) and you can find my material on GitHub and SlideShare related to my “Telemetry with Windows 10 IoT : from Raspberry Pi2 to Event Hubs” session.

Of course, who follows me knows that I’m currently developing and updating (since about 2 years) the M2Mqtt library as part of the Eclipse Paho project (starting from October 2014) as the main MQTT client implementation for all .Net platforms (full, compact and micro) and WinRT (Windows 8.1 and Windows Phone 8.1) but … what about Windows 10 and Windows IoT Core ?

Don’t worry guys ! The last stable release (4.2.0.0) in its WinRT version works great on the new awesome Microsoft OS from the PC to the Raspberry Pi 2 thanks to the UWP (Universal Windows Platform) programming model. You can use the last Nuget package for sending telemetry data from your embedded system, your PC or smartphone writing only one application !

In the next few lines, I’ll describe how to use this package to send simple temperature value from my home. I’d like only to show how simple is to use the library and give you the right tools to create you IoT solution MQTT based for a Windows IoT Core system !

The hardware : Raspberry Pi2 and TMP102

Of course, it’s possible to develop an Internet of Things solution without an embedded system. Yes, of course ! I’m not crazy but I don’t want to discuss about this scenario (IoT is very huge you could think) during this article that will be focused on using an embedded hardware like the Raspberry Pi 2 with latest release of Windows IoT Core. Online, you can find a lot of article and official documentation on how to setup your board so I can consider this step already done for you !

Regarding the telemetry scenario … what is the easiest data to send … a temperature value of course ! 🙂

For this, I chose a temperature sensor I already had … the TMP102 from Texas Instruments in the related breakout board from Sparkfun. The connections to the Pi2 are so simple because the sensor supports I2C protocol and referring to the Pi 2 pinout we have (left Pi 2 pin and right TMP102 pin) :

  • PIN 1 (3.3V PWR) –> V+ (power)
  • PIN 3 (I2C1 SDA) –> SDA (I2C data)
  • PIN 5 (I2C1 SCL) –> SCL (I2C clock)
  • PIN 6 (GND) –> GND (ground)
  • PIN 9 (GND) –> ADD0 (for I2C address selection)

You can leave the ALT pin of TMP102 not connected to Pi 2 because it’s for alert notifications from the sensor if temperature is higher or lower than a specific threshold. If you chose to use it, you need to connect it to a GPIO pin on the Pi 2 to receive an interrupt (Gpio Value Changed event) on alert.

Create UWP app and add the M2Mqtt package

Using Visual Studio 2015, it’s so simple to create a new UWP application from “File –> New Project –> Windows –> Universal” as showed in the following picture.

01_pi2mqtt

To develop on Raspberry Pi 2 and using all IoT features like Gpio pins, I2C and SPI devices, you need to add the Windows IoT Extensions for the UWP under references and selecting “Select Universal –> Windows –> Extensions”.

02_pi2mqtt

Now you are able to interact with all your Pi 2 pins and sensors you can connect to it but what about the M2Mqtt library for sending data to the Cloud using MQTT protocol. As I said, the library is available as Nuget package without need to recompile it from source code (it’s open source ! you can do it if you want !). Adding the package to your project is so simple under “References -> Manage NuGet Packages”, search for “M2Mqtt” and click on “Install”.

03_pi2mqtt

At the end of the installation you can find the package under the “Solution Explorer” window.

04_pi2mqtt

Time to code !

Now it’s time to code our application, reading from the temperature sensor and publish data to an MQTT broker. The first step is to interact with TMP102 sensor to get the temperature value. To do this I already wrote the driver you can find on GitHub here and include in your project.

To be much simpler as possible I wrote my code in the MainPage constructor to instantiate the TMP102 class and the MqttClient class from M2Mqtt library and to launch a task that reads the temperature and send it to the broker every 5 seconds. Of course, you have to write better code (for example using a Backgroud IoT Task instead of an UWP app) … but this sample is useful to show you how to integrate M2Mqtt in your project and how it’s simple to use it.

public sealed partial class MainPage : Page
{
    private TMP102 tmp102;
    private MqttClient client;

    public MainPage()
    {
         this.InitializeComponent();

         this.tmp102 = new TMP102();
         this.client = new MqttClient("test.mosquitto.org");

         this.client.Connect(Guid.NewGuid().ToString());

         Task.Run(async () =&gt;
         {
              bool isOpened = await this.tmp102.OpenAsync();

              while (true)
              {
                   float temperature = this.tmp102.Temperature();
                   string json = "{ temp : " + temperature + " }";

                   this.client.Publish("/pi2mqtt/temp", Encoding.UTF8.GetBytes(json));

                   await Task.Delay(5000);
              }
         });
    }
}

To simplify the solution I decided to use the publically available Mosquitto broker at this address : test.mosquitto.org

The above example is very simple. First you have to create TMP102 and MqttClient instances (for the last one you need to specify the broker host name). The second step is to connect to the MQTT broker and start the task for reading and sending data. Inside the task, every 5 seconds, the temperature value is read from the TMP102 instance and it’s published to the broker on the “/pi2mqtt/temp” topic in JSON format (as payload of the MQTT message).

Now that your project is sending value to the broker, the simplest way to see you are publishing data is to use another MQTT client, for example the mosquitto client and subscribe to the same topic to receive data. After installing mosquitto, you can start the mosquitto subscriber client in the following way :

mosquitto_sub -h test.mosquitto.org -t /pi2mqtt/temp -q 0 -d

Now … you should see data received from you Raspberry Pi 2 !

05_pi2mqtt

Conclusion

Of course, it isn’t an article to deep into Windows IoT Core, MQTT protocol, M2Mqtt library and so on. I’d like only to show how you have an “out of box” way (as the M2Mqtt library) to integrate your embedded system into a huge MQTT solution and you can develop it using Windows IoT Core as OS or in general your Windows 10 system !

M2Mqtt on MQTT encyclopedia and me on DotNetPodcast with “IoT end to end”

Today, I’d like to highlight some stuff …

mqtt_client_encyclopedia

First, my guest article on the M2Mqtt project in the “MQTT Client Library Encyclopedia” blog post series hosted on HiveMQ web site (thanks to the team for asking me to write this entry). It’s great for this library to be part of this “encyclopedia” that will host all main MQTT client implementations in all different languages and technologies in the next weeks. M2Mqtt represents the Microsoft based one developed in C# and available for all .Net platforms (full, compact and micro) and WinRT (Windows 8.1, Windows Phone 8.1 and Windows 10).

iot_end_to_end

Second, as always I want to thanks the DotNetPodcast team for registering a new podcast, titled “IoT end to end”, on building an Internet of Things solution from devices to the cloud using all Microsoft Azure related technolgies. I described all the main steps that builds an IoT scenario speaking about devices, gateway (in the field and in the cloud), protocols, data ingestion with Azure Event Hubs, analysis in real time with Azure Stream Analytics, data storage and presentation to the user. Of course I spoke a little about the future with IoT suite and the new IoT Hub service. I’m sorry for all my non italian friends … but the podcast is in italian. If you like you can find all my podcasts (only two for now) on the dedicated page of my blog.

AMQP message accepted : encoding on the wire

We saw the AMQP builtin type system, we sent a message to the broker …. now, how is encoded the receive confirmation from the broker itself ?

In this case, the so called performative is the disposition (format code 0x15, page 71 on official specification) that has some fields all filled in a lot of cases.

amqp_disposition

Of course it has the descriptor 0x00 0x53 and 0x15 as format code and it’s a list 0xC0 with 0x09 bytes distributed on 0x05 fields.

The first field is role that defines the role of the peer that could be sender or receiver. In this case the value is 0x41 that is the “true” value for the boolean type (page 26 of official specification). Why ? What’s the relationship between role and boolean ? The role type is defined as a restricted type that has boolean has source. You can see it like an inheritance because role derives from the existing boolean type restricting and/or changing the mean of the related values using choice (page 73 of official specification). In this case we have that sender is “false” and receiver is “true”. In our case we have 0x41 (true) so the role is the receiver and it’s true because we send a message to the broker (to a queue) and it means that on the other side of the link, the peer is the receiver.

After role, we have first & last fields. Each message sent by the sender has a “delivery-id”. When the receiver sends the disposition to the sender to confirm received message, it needs to set the “delivery-id” of the message itself. It’s possible to define a “delivery-ids” interval so that the receiver can confirm more received messages with a single disposition. The two limits are defined by first and last fields that are of type delivery-number (page 74 of official specification) that is a restricted type with sequence-no as base (it’s a serial number as defined in RFC-1982). In our example the receiver has to confirm only one message so first is 0 and last is null. As you can see, the 0 value is expressed with the byte 0x43 that is a specific and well defined byte used to describe a 0 value (uint0, page 26 of official specification).

Next field is settled that defines the delivery status of the message on the endpoint/peer. This value is also related to the QoS (Quality of Service) and it acts as an aknowledge on messages exchanged. For example, the settled field is in the transfer disposition too; a sender can set it to true and it means to apply a “fire and forget” QoS (the sender sends the message and it isn’t interested to have an acknowledge by the receiver). If the sender set it to false, it expects to receive a disposition with settled to true from receiver (it’s a “at least once” QoS because if the disposition message is lost, the sender re-send the message). Of course, AMQP supports “exactly once” QoS (it isn’t supported by Azure).

In this case its value is 0x41 (true) because it’s defined as a boolean type;

The last field is state that defines the delivery outcome with following possible values :

  • accepted : the message was processed by the receiver;
  • rejected :  indicates an invalid and unprocessable message;
  • released : the message was not (and will not be) processed;
  • modified : message was modified, but not processed;

Due to the above outcomes, a message can go through following state transitions :

amqp_message_states

In our example the message was accepted and the field has the descriptor 0x00 0x53 0x24 where the 0x24 format code defines the accepted outcome (page 89 of official specification) that is a list. In this case the list is empty and it’s defined by the byte 0x45 used to describe an “empty list”.

AMQP on the wire : messages content framing

After the previous post on the AMQP builtin type system, I’d like to show you a different sample frame strictly related to the sending messages. For starting, I used the SASL mechanisms frame as example to understand the encoding/decoding system just because it’s one of the first exchanged frames in the protocol but every day we use AMQP to send messages so understand their encoding is much more important.

An AMQP message on the wire

Consider the following source code that uses AMQP .Net Lite library to send a simple message :

Connection connection = new Connection(address);
Session session = new Session(connection);
SenderLink sender = new SenderLink(session, "sender-" + testName, "q1");

Message message = new Message("Hello");
message.Properties = new Properties();
message.Properties.MessageId = Guid.NewGuid().ToString();
message.ApplicationProperties = new ApplicationProperties();
message.ApplicationProperties["prop1"] = 1; 
message.ApplicationProperties["prop2"] = "value";

sender.Send(message, 60000);

In the above code, we are interested in the encoding and framing used in the AMQP protocol to send the Message object on the wire. As reported on the official specification (page 38), each frame has a format as showed in the following picture (we already saw this format for the SASL frames with some “little” changes).

amqp_frame

As we can see, the real body starts with the so called performative; you can think about it as an “operation” executed against the AMQP broker (opening the connection or the session, attaching the link, transfering a message and so on).

amqp_message_01

In this case, we have the transfer performative for sending messages and the encoded frame is much more complex than the first one (SASL mechanisms).

The objective of this article is to understand how the builtin type system is used to encode the three main parts of an AMQP message.

amqp_message_format

From the above picture we’ll consider :

  • Message properties : system properties well defined by the AMQP specification and used by the broker for manipulation, routing and so on;
  • Application properties : user defined properties to carry data without using the following payload;
  • Application data : user defined payload of the message;

The content of all these parts is encoded using the builtin type system. Starting from the “Message-Properties” section we have the descriptor (0x00 0x53 0x73) where the 0x73 code defines the “AMQP properties list” (page 84 of official specification) that is a list (format code 0xC0) of 39 bytes (0x27) with only one element (0x01). This element is an UTF8 encoded string (0xA1) with size of 36 characters expressed with only one byte (0x24). This string is the “Message-Id” we set in our source code (it’s the autogenerated GUID value). You can see that there is no information to describe that this field is the “Message-Id” field but we know that because it’s the first field of the composite type “AMQP properties list”. If the “Message-Id” field weren’t explicitly set by code, it would be null but encoded inside the properties list itself. This means that all missing fields between two defined fields are encoded as null but inserted inside the list (to guarantee the fields sequence); if all subsequent fields (until the end of the list) aren’t set, they aren’t encoded inside the frame (to avoid a waste of space).

amqp_msg_properties

The “Application-Properties” section starts with its related descriptor (0x00 0x53 0x74) where the 0x74 code defines the “Application properties map” (page 86 of official specification) that is a map (format code 0xC1) of 24 bytes (0x18) with four elements (0x04). A map is a compound type in the AMQP builtin type system we didn’t see yet; it’s a collection with key-value pairs each encoded as defined in the specification.

amqp_map

In our example, we have two pairs (“prop1” = 1 and “prop2” = “value”) so four elements in the map. The name of the first key “prop1” is encoded as a UTF8 string (code 0xA1) with 0x05 a size (five characters for “prop1”) and the related value that is the string “value” is encoded in the same way. Then there is the name of the second key “prop2” and the related value 0x01 that is an integral type expressed with only one byte (format code 0x54, page 27 on official specification).

amqp_app_properties

The last part of the message is the data section (the “Hello” string) that has the descriptor (0x00 0x53 0x77) where the code 0x77 defines an “AMQP value” (page 86 of official specification) that in this case is an UTF8 string (0xA1) with 0x05 characters.

amqp_value

Changing the data as raw binary

The interesting thing is that the data part of an AMQP message could be encoded as an “AMQP value” (as previous), as an “AMQP sequence” (of values) or as pure “Binary” data.

Consider the previous example but with following changes related to the creation of Message instance.

Message message = new Message()
{
    BodySection = new Data() { Binary = Encoding.UTF8.GetBytes("Hello") }
};

Using the BodySection as Data object and the related Binary field the encoding of “Hello” string on the wire is defined as binary (0xA0) with 0x05 following bytes.

amqp_binary_data

Encoding the data in the following way is an advantage for the consumer of the message that needs to understand pure binary data and not “AMQP value” encoding.

Conclusion

The AMQP message format seems to be more complex than any other protocol but deeping into it …. it’s seems to be simpler.

Of course, there is another part in the message we didn’t consider named the “Message Annotations” that are used a lot for customization inside brokers like for Event Hubs inside Microsoft Azure Service Bus.

However, we can’t see them on the wire because we know that Service Bus traffic is encrypted (using SSL/TLS protocol) and the local AMQP broker for testing doesn’t support event hubs for clear reasons 🙂

AMQP protocol : the builtin type system by examples

One of the most interesting features of AMQP protocol is the built in type system that provides a way to represent the AMQP values inside a frame starting from primitive types to custom types (based on our application domain) and composite types (containing fields).

During this article I’ll try to explain in a simple way the main features of the type system but for better understanding you can refer the official AMQP specification available at following link.

AMQP type system : a brief introduction

First of all, any AMQP data is encoded in the following way :

amqp_type

As you can see, it has a constructor followed by untyped bytes. The constructor could be a code to represent a primitive type (named primitive format-code) or a code to represent a described type that is useful to describe custom/complex types in our application domain; a described type consists of a descriptor followed by a primitive format-code.

For example, the encoding for the “Hello world” string is the following :

0xA1 0x0B “Hello World”

where :

  • 0xA1 : it’s the simple constructor with a single byte that represents the primitive format-code for UTF8 encoded strings. This format-code has the higher nibble “A” (named subcategory) that represents a data with variable width defined by a single byte so with size between 0-255 (page 19 on official specification). The lower nibble “1” is named subtype and the entire byte represents the type UTF8 encoded string with a max length of 255 characters (page 29 of official specification);
  • 0x0B : it’s the length of the string (11 bytes). As you can see, the length is represented with a single byte as described by the previous constructor 0xA1;
  • “Hello world” : the UTF8 encoded string represented as AMQP value;

Representing a primitive type like a string is very simple because the constructor is simple itself but describing a custom type is more complex. Using the same example from the official specification, imagine you want to define a custom type “URL” (of course, it’s a string but not all strings are URLs) and to encode an URL as AMQP value. In this case, the constructor doesn’t consist of a single byte (a simple primitive format-code) but it has a descriptor followed by a primitive format-code. The very interesting thing is that the descriptor itself is a valid AMQP value so it’s encoded in the same way ! It means that we need a constructor followed untyped bytes to define a descriptor !

Referring the URL example we have :

0x00 0xA1 0x03 “URL” 0xA1 0x1E “http://example.org/hello-world” 

where :

  • 0x00 : it’s the format-code that defines a descriptor (page 18 of official specification);
  • 0xA1 : it’s the primitive format-code used to define the custom type within the descriptor. Our custom type is defined as the UTF8 encoded string “URL” so a string with max length above 255 characters;
  • 0x03 : it’s the length of the string (3 bytes);
  • “URL” : the UTF8 encoded string used to define the type with the whole descriptor;
  • 0xA1 : it’s the primitive format-code that is part of the constructor for a described type and follows the above descriptor. In this case it represents an UTF8 encoded string with max length of 255 characters;
  • 0x1E : it’s the length of the string (30 bytes);
  • http://example.org/hello-world ” : the UTF8 encoded string represented as AMQP value;

In this way we don’t represent a simple string but a more specific string because we have defined the custom type “URL” followed by the URL value.

Of course, the AMQP type system is more complex and provides a lot of simple and complex type from scalar (integral numbers, booleans, strings and so on) to collections (arrays, lists and maps). To deep into them you can refer to the official AMQP specification.

Inside a sample frame : SASL mechanisms list

After a quick introduction to the AMQP type system, we can try to analyze a real frame sniffing the AMQP traffic with Wireshark who provides a specific AMQP protocol analyzer. Of course, we can’t collect this traffic if our client connects to Azure Service Bus (queues, topics/subscriptions or event hubs) because this traffic is encrypted with an SSL/TLS connection. For this reason, to have plain (not encrypted) traffic, I used the AMQP broker built in the AMQP .Net Lite library and my Azure SB Lite as client.

01_wireshark_amqp

After TCP handshaking between client and server and after the protocol header exchange, there is the SASL traffic that is used to define the security and authentication layer that will be used for data exchange. The first is the sasl-mechanisms list frame sent from the server to the client to announce the supported authentication mechanisms (for more information, starting from page 113 of official specification).

Following a screenshot of such a frame :

02_sasl_mechanisms_frame

The first 8 bytes are part of the AMQP frame header (page 37 of official specification) as it’s defined by the specification so we have :

  • first 4 bytes (0x00 0x00 0x00 0x1B) represent the entire frame length (which is 27 bytes) as decoded by Wireshark plugin;
  • the 5th byte is the DOFF byte (data offset) which represents where the frame data starts (the starting offeset is 4 * DOFF);
  • the 6th byte is the frame type so what’s the content inside the payload. In this case it’s 0x01 that represents a SASL frame;
  • the last 2 bytes of the header are ignored in the case of a SASL frame. In a more generic AMQP frame they represents the channel and it’s the reason why Wireshark plugin decodes them as channel 0 even if it’s a non sense.

After the AMQP frame header we have the frame body that is encoded using the “awesome” builtin type system.

sasl_frame

The first byte is 0x00 (descriptor format-code), so it means that we have a not simple constructor that has a descriptor inside it for defining a custom type. What’s this custom type we are describing ? It’s the SASL mechanisms list that contains all the authentication mechanisms supported by the server. This descriptor has 0x53 as primitive format-code that indicates a fixed-one value so an integral value (ulong type on page 26 of official specification) with a size expressed by a single byte (0 – 255). The following value 0x40 represents the custom SASL mechanisms list type as defined in the official specification on page 115. As we can see on this page, it’s a composite type which is a list as we can expected and it’s represented by the following 0xC0 byte (page 29 of official specification).

In the AMQP type system, a list is a polymorphic collection (as compound type) that could contain values with different types (ex. an integral, a boolean, an array …) so it’s encoded as follow :

amqp_compound_type

There is the total size of the list in byte (SIZE) and the number of items (COUNT). Each item consists of a constructor and the related data (this is the way for the list to be polymorphic, because each item doesn’t have only the value but the constructor to specify the type too). Returning to our frame we have 0x0E that is the size of the list (14 bytes) followed by 0x01 (the number of items, only one).

But … what’s the single item inside this list ? Of course it starts with the constructor that is represented with 0xE0 and it meas the format-code for an array. In the AMQP type system, an array is a monomorphic collection so that all the items are of the same type. For this reason, it’s encoded in the following way :

amqp_array_type

The encoding is like the list type (SIZE and COUNT) but it has only one constructor to define the single type for all the items inside the collection followed by all the item values themself.

In our frame, we have 0x0B (total size of array in bytes, 11 bytes) and 0x01 (only one element inside the array). After that we have the item constructor represented by 0xB3 format-code that is a “symbol” type (page 29 of the official specification). Symbols are values from a constrained domain; you can think them as enums with a limited number of possible values. The 0xB3 code represents a symbol with a size expressed with 4 bytes, so the following 4 bytes in the frame (0x00 0x00 0x00 0x05) represent a size of 5 bytes for the symbol value.

The symbol value is “PLAIN” and it means that the broker supports only PLAIN authentication method. We can notice that the frame value isn’t encoded as a simple string (primitive format-code 0xA1 we saw previously); a symbol is used because it wants to define an “enum” due to the limited possible values for SASL authentication mechanisms (PLAIN, EXTERNAL, ANONYMOUS and so on).

amqp_sasl_mechanisms_notes

Conclusion

Finally we have decoded the frame and it means we have a list with only one element that is an array of only one element that is the PLAIN symbol !

We could ask why there is a list with an array inside it and not a simple array without external list ! From the official specification (page 22) we can see that a composite type is sent on the wire as a list and it must be always a list. The SASL mechanisms frame is described as a composite type (page 115) so it must be a list.

Inside it we have an array because the broker may support more than one authentication methods and not only one as in this example.

I hope you found this article useful to understand the AMQP type system but you have to consider this blog post as a starting point to deep into the official specification.

Mobile Camp 2015 : telemetria da Windows 10 IoT Core all’Azure Event Hubs

mobile_camp_2015

Anche quest’anno ho avuto il piacere di essere speaker al Mobile Camp organizzato da Microsoft a Napoli; quest’anno l’evento era incentrato su Windows 10 e tutte le novità di Build 2015.

Da buon Microsoft MVP su Windows Embedded ed IoT, la mia sessione era ovviamente dedicata all’embedded ed all’Internet of Things ed in particolare alla Raspberry Pi 2 con Windows 10 IoT Core.

Ovviamente, in una soluzione IoT non può mancare il Cloud con i suoi servizi e nel mio caso non poteva mancare l’Azure Event Hubs come destinazione dei dati (temperatura) trasmessi dalla Raspberry Pi 2 (utilizzando la mia libreria Azure SB Lite).

Per chi fosse interessato, il materiale è disponibile su SlideShare (slide) e su GitHub (il codice sorgente della demo).