Your enterprise containers orchestrator in the cloud: deploying OpenShift on Azure

In a “containerized” world running “cloud native” applications, there is the need for a containers orchestrator as Kubernetes which is the best open source platform providing such features. Furthermore, Red Hat pushed this project to an enterprise-grade level developing OpenShift (on top of Kubernetes) adding more powerful features in order to simplify the applications development pipeline. Today, thanks to this platforms, you can easily move your containerized applications (or microservices based solutions) from an on-premise installation to the cloud without changing anything on the development side; it’s just a matter of having a Kubernetes or OpenShift cluster up and running and moving your container from one side to the other.

As already announced during the latest Red Hat Summit in San Francisco, in order to improve the developers experience, Red Hat and Microsoft, are working side by side for providing a managed OpenShift service (as it is already available for Kubernetes with AKS); you can read more about this effort here.

But … while waiting for having this awesome and really easy solution, what’s the way for deploying an OpenShift cluster on Azure today?

Microsoft is already providing some good documentation for helping you on doing that for both the open source OKD project (Origin Community Distribution of Kubernetes, formerly known as OpenShift Origin) and the Red Hat productized version named OCP (OpenShift Container Platform) if you have a Red Hat subscription. Furthermore, there are two different GitHub repositories which provide templates for deploying both: the openshift-origin and the openshift-container-platform.

In this blog post, I’d like to describe my personal experience about deploying OKD on Azure trying to summarize and explain all the needed steps for having OpenShift up and running in the cloud.

Get the Azure template

First of all, you have to clone the openshift-origin repository, available under the Microsoft GitHub organization, which provides the deployment template (in JSON format) for deploying OKD.

git clone https://github.com/Microsoft/openshift-origin.git

Because the master branch contains the most current release of OKD with experimental items, this may cause instability but includes new things. For this reason, it’s better switching to one of the branches for using one of the stable releases, for example, the 3.9:

git checkout -b release-3.9 origin/release-3.9

This repository contains the ARM (Azure Resource Manager) template file for deploying the needed Azure resources, the related configurable parameters file and the Ansible scripts for preparing the nodes and deploying the OpenShift cluster.

You will come back to use the artifacts provided in this repository later because there are some other steps you need to do as prerequisites for the final deployment.

Generate SSH key

In order to secure access to the OpenShift cluster (to the master node), an SSH key pair is needed (without a passphrase). Once you have finished deploying the cluster, you can always generate new keys that use a passphrase and replace the original ones used during initial deployment. The following command shows how it’s possible using the ssh-keygen tool on Linux and macOS (for doing the same on Windows, follow here).

ssh-keygen -f ~/.ssh/openshift_rsa -t rsa -N ''

The above command will store the SSH key pair, generated using the RSA algorithm (-t rsa) and without a passphrase (-N ”), in the openshift_rsa file.

In order to make the SSH private key available in the Azure cloud, we are going to use the Azure Key Vault service as described in the next step.

Store SSH Private Key in Azure Key Vault

Azure Key Vault is used for storing cryptographic keys and other secrets used by cloud applications and services. It’s useful to create a dedicated resource group for hosting the Key Vault instance (this group is different from the one for deploying the OpenShift cluster).

az group create --name keyvaultgroup --location northeurope

Next step is to create the Key Vault in the above group.

az keyvault create --resource-group keyvaultgroup --name keyvaultopenshiftazure --location northeurope --enabled-for-template-deployment true

The Key Vault name must be globally unique (so not just inside the just created resource group).

Finally, you have to store the SSH key into the Key Vault for making it accessible during the deployment process.

az keyvault secret set --vault-name keyvaultopenshiftazure --name openshiftazuresecret --file ~/.ssh/openshift_rsa

Create an Azure Active Directory service principal

Because OpenShift communicates with Azure, you need to give it the permissions for executing operations and it’s possible creating a service principal under Azure Active Directory.

The first step is to create the resource group where we are going to deploy the OpenShift cluster. The service principal will assign the contributor permissions to this resource group.

az group create --name openshiftazuregroup --location northeurope

Finally, the service principal creation.

az ad sp create-for-rbac --name openshiftazuresp --role Contributor --password mypassword --scopes /subscriptions//resourceGroups/openshiftazuregroup

From the JSON output, it’s important to take a note of the appId field which will be used as aadClientId parameter in the deployment template.

Customize and deploy the Azure template

Back to the openshift-origin GitHub repository, we have cloned before, the azuredeploy.parameters.json file provides all the parameters for the deployment. All the corresponding possible values are defined in the related azuredeploy.json file.

This file defines the deployment template and it describes all the Azure resources that will be deployed for making the OpenShift cluster such as virtual machines for the nodes, virtual networks, storage accounts, network interfaces, load balancers, public IP addresses (for master and infra node) and so on. It also describes the parameters with the related possible values.

Taking a look at the parameters JSON file there are some parameters with a “changeme” value. Of course, it means that we have to assign meaningful values to these parameters because they make the main customizable part of the deployment itself:

• openshiftClusterPrefix: a prefix used for assigning hostnames for all nodes – master, infra and nodes.
• adminUsername: admin username for both operating system login and OpenShift login
• openshiftPassword: password for the OpenShift login
• sshPublicKey: the SSH public key generated in the previous steps (the content of the ~/.ssh/openshift_rsa.pub file)
• keyVaultResourceGroup: the name of the resource group that contains the Key Vault
• keyVaultName: the name of the Key Vault you created
• keyVaultSecret: the Secret Name you used when creating the Secret (that contains the Private Key)
• aadClientId: the Azure Active Directory Client ID you should have noted when the service principal was created
• aadClientSecret: the Azure Active Directory service principal secret/password chosen on creation

Other useful parameters are:

• masterVmSize, infraVmSize and nodeVmSize: size for VMs related to the master, infra and worker nodes
• masterInstanceCount, masterInstanceCount and masterInstanceCount: number of master, infra and worker nodes

All the deployed nodes run CentOS as the operating system.

Instead of replacing the values in the original parameters file, it’s better making a copy and renaming it, for example, as azuredeploy.parameters.local.json.

After configuring the template parameters, it’s possible to deploy the cluster running the following command from inside the GitHub repository directory (for using the contained JSON template file and the related parameters file).

az group deployment create -g openshiftazuregroup --name myopenshiftcluster --template-file azuredeploy.json --parameters @azuredeploy.parameters.local.json

This command could take quite a long time depending on the number of nodes and their size and at the end, it will show the output in JSON format from which the two main fields (in the “outputs” section) are:

• openshift Console Url: it’s the URL of the OpenShift web console you can access using the configured adminUsername and openshiftPassword
• openshift Master SSH: contains information for accessing to the master node via SSH
• openshift Infra Load Balancer FQDN: contains the URL for accessing the infra node

In order to access the OpenShift web console, you can just copy and paste the “openshift Console Url” value in your preferred browser.

In the same way, you can access the OpenShift master node by just copying and pasting the “openshift Master SSH” value in a terminal.

Once on the master node, you can start to use the oc tool for interacting with the OpenShift cluster, for example showing the available nodes as follows.

You can notice that these URLs don’t have really friendly names and it’s because the related variables in the template for defining them are:

"infraLbPublicIpDnsLabel": "[concat('infradns', uniqueString(concat(resourceGroup().id, 'infra')))]",

"openshiftMasterPublicIpDnsLabel": "[concat('masterdns', uniqueString(concat(resourceGroup().id, 'master')))]"

Conclusion

Nowadays, as you can see, following a bunch of simple steps and leveraging the Azure template provided by Microsoft and Red Hat, it’s not so difficult to have an OpenShift cluster up and running in the Azure cloud. If you think about all the resources that need to be deployed (nodes, virtual network, load balancers, network interfaces, storage, …), the process really simplifies the overall deployment. Of course, the developers’ life will be easier when Microsoft and Red Hat will announce the managed OpenShift offer.

Having an OpenShift cluster running in the cloud it’s just the beginning of the fun … so, now that you have it … enjoy developing your containerized applications!

How to learn Kubernetes ? “Kubernetes in action” the answer !

Few months ago I started the Manning Access Early Program (MEAP) for one of the books that I think is the best resource for all the newbies who want to start studying Kubernetes and for all the experts who want to dig into its details and internals.

It’s “Kubernetes in action” written by Marko Luksa , one of my colleagues in Red Hat as software engineer in the Cloud Enablement Team.

First of all, I can guarantee on the author ! Marko has a really deep knowledge of Kubernetes and OpenShift and luckily for us, he decided to write a book about that. He is a very nice person, always available to help you to solve any problem that you are facing with Kube. I was lucky to start working with Marko when the EnMasse project was born.

Speaking about the book, it’s awesome !

After the first part introducing Docker and Kubernetes and the first steps with it, the book moves to the core concepts.

You can find all information about what pods are and how you can deploy containers (so your applications) with them and finally how you can replicate pods. Then how to make applications accessible inside and outside the cluster using services and how to use storage for having a persistence layer for your data shared between pods and always available on restarts. Do you want to know how your application is configurable even with sensitive data (i.e. certificates and credentials), you will find such information in this book !

After covering all these core concepts, the last big part has the objective to bring you more deep information about Kubernetes. First of all, the new StatefulSets feature which allows to deploy stateful applications with their stable identity and stored data across restarts. Then a really interesting look to Kubernetes internals speaking about all the components which made it : etcd, the API server, the controller manager and all the other stuff. Managing resources is another interesting chapter describing how you can request specific resources in terms of CPU and memory for the containers even setting limits on them.

But one of the big advantage of using a container orchestrator like Kubernetes is the possibility to scale your infrastructure based on the load you have against the applications. You will find this information about auto-scaling on CPU utilization or custom metrics !

The final part is enriched by best practices for developing cloud native applications which run on Kubernetes. In order to learn a new technology, knowing how it works is the main part but it’s more useful having examples and patterns provided by expert people like Marko.

Finally, the book ends explaining how it’s possible to extend Kubernetes defining new components and custom API objects so showing a powerful feature like its extensibility.

In conclusion, I think that this book deserves to be read and to be part of your books collection even because after reading it you’ll become expert on two technologies for developing cloud native applications : not only Kubernetes but OpenShift as well ! 🙂

Eclipse Hono : “Connect. Command. Control” … even on OpenShift !

The Eclipse Foundation is the main open source community which has a great focus on the Internet of Things world and the related Eclipse IoT ecosystem involves a lot of different projects and partners like Red Hat, Bosch, Eurotech, IBM and many more. Recently, publishing this white paper, they showed a full stack with all the available tools for building a complete IoT solution from devices to the Cloud through the gateways.

In relation to the Cloud side, one of the main problems in the IoT world is the ability to handle millions of connections from devices (and gateways) in the field, their registration for managing authentication and authorisation and, last but not least, the ingestion of the data received from them like telemetry or events. Finally, the last point is related to control these devices sending them commands in order to execute actions in the environment around them or upgrading their software and configuration.

The Eclipse Hono™ project is the answer to these problem !

The APIs

From the official web site, we can read :

Eclipse Hono™ provides remote service interfaces for connecting large numbers of IoT devices to a back end and interacting with them in a uniform way regardless of the device communication protocol.

The mantra from the landing page of the web site project is “Connect. Command. Control” which is made a reality through a well defined API for :

• Registration : for handling the requests related to the registration (so creation) of a new device so that it can be authorised to connect. It’s also possible retrieving information about registered devices or delete them;
• Telemetry : for the ingestion of a large volume of data from devices (i.e sensors) available for analysis to the backend applications;
• Event : for receiving specific events (i.e. alarms, notification, …) from devices for making decision on the Cloud side. This API is quite similar to a telemetry path but it uses a “different” channel in order to avoid such events going through the overwhelmed telemetry one;
• Command & Control : for sending commands to the devices (from a backend application) for executing operations and actions on the field (receiving the related response) and/or upgrading local software and configuration;

All the above APIs are accessible through the AMQP 1.0 protocol (the only standardised AMQP version!) and it means that they are defined in terms of addresses on which devices need to connect for interacting with the system and the properties/content of the messages; of course, it’s true not only for devices but even for business and backend applications which can receive data from devices or send them commands. In any case, it doesn’t mean that devices which aren’t able to speak such protocol can’t connect but they can leverage on the protocol adapters provided by Hono; the current implementation provides an MQTT and HTTP REST adapter.

All these APIs are though in order to allow multi-tenancy so that using a single deployment, it’s possible to handle channels for different tenants so that each of them can’t see data or exchanged messages from the others.

The Architecture

The main components which build the Eclipse Hono™ architecture are :

1. Protocol Adapters : these components adapt a device protocol to the first citizens protocol used in Hono, the AMQP 1.0. Today, an MQTT and HTT REST adapters are provided out of box but thanks to the available interfaces, the user can develop a new adapter even for some custom protocols;
2. Hono Server : this is the main component to which devices can connect directly through AMQP 1.0 or through the protocol adapters. It’s in charge to expose the APIs in terms of endpoints and handling the authentication and authorisation of devices;
3. Qpid Dispatch Router : this is an AMQP 1.0 router, part of the Apache Qpid project, which provides the underlying infrastructure for handling millions of connections from devices in the fields. The simpler deployment can use only one router but in order to guarantee reliability and high volume ingestion, a router network should be used;
4. ActiveMQ Artemis : this is the broker mainly used for handling command and control API so for storing commands which have to be delivered to devices;

While the devices connect directly to the Hono Server (or through protocol adapters), the backend applications connect to the Qpid Dispatch Router leveraging on direct addressing for receiving telemetry data (if no application is online, no devices are able to send data) or sending commands (the queus backed in the broker are reachable through the router).

The running environment

All the artifacts from the project are provided as Docker images for each of the above components that can run using Docker Compose (the Docker Swarm support will be available soon) or using a more focused Cloud platform like OpenShift (compatible with Kubernetes deployment as well).

Regarding the OpenShift deployment, the building process of the Hono project provides a bunch of YAML files which describe the objects to deploy like deployment configurations, services, persistent volumes and related claims. All the need is having an OpenShift platform instance accessible and deploy such objects for having Eclipse Hono™ running in a full featured Cloud environment with all the related scaling capabilities.

The example deployment is based on four pods, one for each of the main components so there is the router pod, the Hono Server pod and one pod for each protocol adapter; of course if you need the command & control path, even the broker pod need to be deployed.

In order to try it, an OpenShift Origin instance can be used on a local PC for deploying the entire Hono solution; for example, the above picture is related to my tests where you can see all the pods running on OpenShift (left side) with simulated devices that interact using MQTT and HTTP REST (on the right side).

The documentation which describes the steps for having such a deployment is available on the official web site here.

So what are you waiting for ? Give it a try !

Conclusion

In my mind every IoT solution should be made of three different layers (a little bit different from the Eclipse vision) : the devices/gateways, the connectivity layer and the service layer.

While the Eclipse Kura project fits in the gateways layer and the Eclipse Kapua in the service layer, Eclipse Hono is the glue between them in order to handle the connections from devices and making available their data to the backend services (and vice versa in the opposite direction for command and control). Thanks to the API standardisation  and the usage of a standard protocol like AMQP 1.0, Hono can be used for connecting any kind of devices with any kind of services; of course leveraging on a these three project has the big advantage of having big companies working on them, mainly Red Hat, Bosch and Eurotech.

Finally, the solution is always the same …. open source and collaboration ! 😉