Internet of Things (IoT) development experience



1. RTSoft GmbH Internet of Things (IoT) development experience

RTSoft GmbH Software Development Center (SDC) has started to look at and perform first investigation steps in Internet of Things area in 2012. At that time even term “IoT” itself has not been widely introduced, “Machine-to-Machine” (M2M) one was used instead. SDC has started its “IoT life” from a proof of concept project, which main goal was to demonstrate capabilities of the new Kontron GmbH FRI2 M2M gateway in action (video surveillance system). Despite this limited goal, the final solution has included such necessary high-level infrastructure components as edge and enterprise tiers. Of course, there were no complex IoT frameworks yet at the time, and each layer was developed without them, using the following technologies: OSGi-based software for gateway itself, Mosquitto MQTT broker as “cloud” layer, Xamarin mobile end-user applications.

Pic.1.1

Since then RTSoft has created a number of IoT applications in building management, advanced power grid monitoring and management, industrial manufacturing, public transportation and many others; and we gained a firm understanding that the best project output can be achieved through the broad open-source IoT frameworks usage, which, fortunately are now widely used in production and quickly innovating.

2. Importance and benefits of open-source development for IoT projects implementation

Nowadays, there is a strongly proven fact for all software developers – open source products are great instruments, which could help a lot for almost any real commercial project you could face in practice. Great slogan for such idea is – “Open-source software provides digital building blocks for creating almost anything we can imagine”. Evident advantages of open-source technologies are:

  • They are free to use and modify;

  • They have great interoperability (there are sets of open standards, which, together with open-source frameworks themselves, constitute a whole ecosystem);

  • They usually have extensive and stable engineering communities, which provide constant quality, support and evolution;

  • Faster time to market, because open source solutions are openly available and can be explored for free, it's often much faster to investigate options and get solutions off the ground.


Pic.2.1

Of course, IoT domain is not an exception (and may be even mostly saturated open-source domain – see Pic.2.1). On the Edge tier level one can find a number of extremely useful Eclipse IoT projects like Kura, MiniFi etc. They are very useful in the integration, of data flow coming from BLE, ZigBEE, WiFi, and wires from sensors, actuators to IoT gateways. These IoT projects are enabling common API for device fleet management. They are also facilitating communication with Platform tier by the means of MQTT, AMQP and other IoT-oriented data exchange protocols.

On platform IoT tier developers could also find a set of various open-source solutions (in message management, storage and monitoring areas) such as Apache Kafka, Cassandra, ActiveMQ, Prometheus, Apache NiFi, Kapua, HawkBit, etc. in order to process and store data and communicate with the higher abstraction level.

Finally, the Enterprise tier level is usually comprising cloud deployment, big data processing and business intelligence tasks. Utilizing Lumify.io, Thinger.io, Grafana, Hadoop and other products is a very common approach for these tasks.

However, practical experience usually shows one common drawback of open-source products usage for IoT domain – the so-called “narrow specialization” of most open-source solutions. It means, in practice, that you always have to select the best (according to your project criteria) option from a set of available ones, and combine them somehow for each real project development. In the other words, there is usually no single open-source IoT platform covering all required practical aspects of a project. This drawback appears not only for separate IoT infrastructure tiers, but even inside them (especially for platform one). So, it was quite reasonable came to idea about necessity to combine and merge several open-source IoT platforms, to have a complex solution, which will be able to cover at least one whole IoT tier.

3. RTSoft GmbH IoT management service (RITMS) at glance

3.1. RITMS high-level structure

Pic.3.1

Following the conclusion about real necessity to combine available IoT frameworks, RTSoft has made some investigations in this area. It was initially obvious for the team, that the most effective place where we need to have a complex solution is exactly the platform tier. Why? Just because of the other tiers’ features: for the edge one, due to its nature - a lot of varied real projects technical requirements - it is really difficult to provide universal solution, and for the enterprise tier there are many specific business value requirements, which make a standardization difficult. So, the platform tier is a subject of our solution, especially, taking into account, it is a real core of any complex IoT project.

First goal, definitely, was to choose the open-source components to be merged with each other. What were criteria here? Just think about the complex solution again. Platform tier is in the middle of IoT infrastructure, so, it would be great to have an easy way to collaborate with adjacent tiers, exactly:

  • To have an ability to receive field data easy way, using some kind of compatible platform on edge tier;

  • Provide an ability to easily create enterprise applications, at the expense of hiding of IoT data processing issues, and let developers to be concentrated on business value aspects.

After the investigation phase, the following components to be merged have been selected: Enmasse.IO (IoT messaging backbone), which is based on Hono platform from Eclipse community, and Orion from FIWARE consortium. They are greatly satisfying the general criteria, listed above. Eclipse Hono (Enmasse.IO uses this as a core) is seamlessly integrated with edge tier Eclipse Kura platform. By-turn, Orion context broker from FIWARE is specially designed to provide normalized and structured data context for IoT enterprise tier applications. Of course, there are many other specific benefits for the each of selected platforms (see chapter 4 below).

So, the choice has been made, and the only issue to create a complex solution is to find a way, how to merge chosen components together. Fortunately, there is official way to do it – FIWARE Orion adapter technology, which has been used to create RTSoft platforms connector module. As a result, the combination of three components – two open-source platforms and one proprietary module – provides a powerful complex solution (see Pic.3.1 above) to facilitate the IoT projects platform tier development.

3.2. RITMS and Kubernetes

Kubernetes (K8s) - https://kubernetes.io - is an open-source system for automating deployment, scaling, and management of containerized applications. It is also called as container orchestration system. The main K8s intention is to group containers that make up an application into logical units for easy management and discovery. K8s platform gives developers the freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and letting effortlessly move workloads to where it is required.

Of course, one of the main advantages of Kubernetes usage is an ability to dramatically decrease the system administration work. Development team just prepare workable application containers, and Kubernetes configuration to be run on master nodes. Thus, deployment, support and upgrade processes are mostly up to developers themselves, and administrator role is just to support the K8s environment.

Kubernetes environment is also able to provide both horizontal and vertical scaling (autoscaling). Depends on current actual workload parameters, Kubernetes can automatically increase or decrease a cluster size, reallocate required cluster resources, start or stop application instances, if need.

Summary of the main Kubernetes benefits are listed below here (see Pic.3.2.1):


  • Service discovery - is the process of figuring out how to connect to a service. To get many of the benefits of containers and cloud-native applications, you need to remove configuration from your container images so you can use the same container image in all environments;

  • Basic invocation. Applications running inside containers can be accessed through Ingress access — in other words, routes from the outside world to the service you are exposing;

  • Elasticity - is solved in Kubernetes by using ReplicaSets (which used to be called Replication Controllers). Just like most configurations for Kubernetes, a ReplicaSet is a way to reconcile a desired state: you tell Kubernetes what state the system should be in and Kubernetes figures out how to make it so;

  • Logging. Since your Kubernetes cluster can and will run several replicas of your containerized application, it’s important that you aggregate these logs so they can be viewed in one place;

  • Monitoring. Using Prometeus for it – it is an open-source monitoring system that includes time series database. It can be used for storing and querying metrics, alerting, and using visualizations to gain insights into your systems;

  • Build and deployment pipelines - CI/CD are often cited as pillars of successful software development and DevOps practices. No software should be deployed into production without a CI/CD pipeline, and Kubernetes supports all these processes;

  • Resilience - While Kubernetes provides resilience options for the cluster itself, it can also help the application be resilient by providing PersistentVolumes that support replicated volumes;

  • Authentication - aims to enhance the security of microservices and their communication without requiring service code changes. It is responsible for: providing each service with a strong identity that represents its role to enable interoperability across clusters and clouds, securing service-to-service communication and end user-to-service communication, providing a key management system to automate key and certificate generation, distribution, rotation, and revocation;

  • Tracing - applications can be configured to collect trace spans using Zipkin or Jaeger.

RITMS utilizes Kubernetes as wide as possible, to use all brilliant benefits, which are listed above. Besides Enmasse.IO, which is totally integrated with Kubernetes by design, another RITMS components – FIWARE Orion context broker and RTSoft platforms connector are both used as Kubernetes services as well.

4. RITMS internals. Components

As it is shown above (Chapter 3.1, RITMS high-level structure), RITMS solution is a combination of two powerful open-source IoT frameworks, and special connector module to merge them to each other. Detailed descriptions of these components are provided below.

4.1. Enmasse.IO – IoT messaging backbone platform


Pic.4.1.1

The first of open-source IoT components selected for RITMS usage is Enmasse.IO (https://enmasse.io/), which main goal, in short, is to hide complexities of messaging infrastructure from on-top solution components.

In more details, EnMasse.IO is an open source multitenant, self-service messaging system for Kubernetes. It can run on your own infrastructure or in the cloud, and simplifies managing the messaging infrastructure for your organization. EnMasse.IO can be used to move your messaging infrastructure to the cloud without depending on a specific cloud provider, building a scalable messaging backbone for IoT (which is exactly a RITMS use case), or just as a cloud-ready version of a message broker.

The core part of Enmasse.IO is well-known Eclipse Hono, which supports multiple IoT protocols, device and tenant management. Eclipse Hono usage also provides a seamless integration with edge tier Eclipse Kura platform, which is a real huge benefit to build complex IoT solutions.

High-level architecture scheme of Enmasse.IO platform is provided on Pic.4.1.1 above. There are 3 main parts of the system there:

  • Micro-services part;

  • AMQP network part;

  • Monitoring part.

Micro-services part consists of two main components:

  • Protocols adapters. Each one covers a specific protocol and is responsible for the authentication of devices by querying the device registry. There are following adapters, provided out-of-the-box: HTTP, MQTT, Eclipse Kura gateways;

  • Device registry item here is responsible for the registration and activation of devices belonging to a tenant, and the provisioning of credentials for devices;

  • Auth Server implements the construction of security tokens to support the authentication of a client;

  • Hono messaging which is intended for the messaging APIs for telemetry and event messages, verification of compliance with the APIs and the forwarding of messages to the AMQP Network.

AMQP network - decouples Enmasse.IO microservices from on-top applications. It provides a scalable messaging infrastructure that supports different communication patterns with various quality of service levels. AMQP is one of very few messaging protocols that is truly symmetrical. It does not require a broker as an intermediary in the message exchange process. It supports both “store and forward” (Artemis broker) and direct messaging (Apache QPID) communication patterns. The symmetrical nature of AMQP allows Hono to use different communication semantics and different messaging components to handle various flows of messages.

Monitoring part – microservices and AMQP network components report different kinds of metrics to and Prometeus data storage, which can be viewed using the Grafana dashboards provided.

Thus, the summary of Enmasse.IO platform advantages could be provided as following list:

  • Out-of-the-box support of most popular IoT messaging patterns (request/response, publish/subscribe, events);

  • Out-of-the-box support for most popular IoT protocols (HTTP, MQTT, Sigfox, LoRaWAN;

  • Ability to easily support any other IoT protocols (new adapters creation);

  • Good scalability from small to large deployments;

  • Built-in authorization and authentication abilities, TLS support with automated certificate management;

  • Built-in monitoring abilities (by using the built-in console or cloud-native tooling);

  • Built-in multitenancy;

  • Easy and flexible deployment process due to Kubernetes container technology usage.

However, there is one drawback of Enmasse.IO, which potentially limits this platform alone usage for real complex IoT projects. The meaning of this drawback could be formulated in a nutshell as: Enmasse.IO is a brilliant technological platform to simplify IoT systems connectivity, messaging and data storing, but it is not initially designed to simplify collaboration with enterprise tier applications, which forces such applications to be more complex for development and future support.

4.2. FIWARE Orion – IoT context broker

4.2.1. General description

The second open-source component of RITMS is FIWARE Orion context broker platform (https://fiware-orion.readthedocs.io/en/master/). Its main intention is to allow developers to manage the entire lifecycle of context information including updates, queries, registrations and subscriptions. From one hand, Context broker interacts with context producer software (which provide sensor information) and, on the other hand, a context consumer software (which processes that information, analyzes and displays – see example on Pic.4.2.1.1 below).


Pic.4.2.1.1

In more details, FIWARE Orion is an NGSIv2 server implementation to manage context information (see Pic.4.2.1.2 below). Using it, developers are able to create context elements and manage them through updates and queries. In addition, it is possible to subscribe to context information so when some condition occurs (e.g. the context elements have changed) a corresponded notification will be sent out.

Pic.4.2.1.2

Orion context broker internal data representation requires so-called “data models”, which assume “entity-attribute” model (see Pic.4.2.1.3 below).

Pic.4.2.1.3

Orion context broker implements a simple multitenant/multiservice model based and logical database separation, to ease service/tenant -based authorization policies provided by other FIWARE components or third-party software. Multitenant/multiservice ensures that the entities/attributes/subscriptions of one service/tenant are "invisible" to other services/tenants.

The summary of FIWARE Orion context broker platform advantages could be provided as following list:

  • Existence of powerful built-in query language to filter and sort data by different criteria. This language can also be used to subscribe to system events;

  • Built-in multitenancy;

  • FIWARE is really wide consortium, which provides not software solutions only, but a lot of other useful options, such as training centers, standards committee, ready-to-use cloud hardware infrastructure (FIWARE Lab facilities). FIWARE activities are approved and supported by European Commission;

  • Out-of-the-box data models, based on a huge past experience, already adopted for various domain areas (smart city, smart energy, smart agriculture etc.);

Orion context broker platform, due to its nature – providing normalized and domain-oriented structured information outside, is a great choice to be used together with enterprise tier end-customer applications.

As for drawbacks of Orion context broker, the quite evident one is – absence of the easy out-of-the-box transparent way to connect to lowest IoT layers (Edge tier). This kind of work is assumed to be done by development team, who uses Orion as a tool, which increases development efforts there.

4.2.2. FIWARE Lab: real-world experimental facilities

One of the specific FIWARE Orion platform benefits is the free usage of pre-existing hardware infrastructure. It is easily possible to set up cloud virtual infrastructure using FIWARE Lab Cloud facilities. In general, FIWARE Lab is the non-commercial sandbox environment of the FIWARE Community. It is deployed over a geographically distributed network of federated FIWARE Lab nodes (see Pic.4.2.2.2 as example). Each FIWARE Lab node maps to one, or a network of, data-centers on top of which an OpenStack instance has been deployed, federated and configured as a FIWARE Lab node (Cloud region) operated by a specific organization.


Pic.4.2.2.1

Pic.4.2.2.2

What, generally, FIWARE Lab infrastructure provides to development teams and entrepreneurs:

  • · Develop once for a large market;

  • · Meet potential customers easy way;

  • · Marketing and promotion abilities;

  • · Ability to test with real data and end-customers;

  • · Simple powerful APIs that accelerates product development.

  • From the technical point of view, developers could utilize the following FIWARE Lab features (see Pic.4.2.2.1 above):

  • · Account: Manages identity and organizations; provides authentication and authorization for other services (using OpenStack Keystone);

  • · Compute: Manages the lifecycle of compute instances. Responsibilities include spawning, scheduling and decommissioning of VMs (using OpenStack Nova);

  • · Network: Enable Network-Connectivity-as-a-Service for other services, e.g. Compute, (OpenStack Neutron);

  • · Storage:

  • - Persistent block storage for running compute instances (OpenStack Cinder);

  • - Stores and retrieves arbitrary unstructured data object and provide storage for other services, e.g. Image, (OpenStack Swift).

RITMS solution contains FIWARE Orion context broker as one of its core components, so all abilities above become available for RITMS users as well.

4.3. RITMS platforms connector plug-in

Pic.4.3.1

As it was mentioned above, RTSoft own know-how inside RITMS solution is platforms connector module development. The main goal here is to translate Enmasse.IO output data flow (AMQP protocol) to FIWARE Orion input one, which assumes NGSI protocol. Orion provides a reference architecture on how to develop so-called “agents”, which are exactly bridges between some external IoT protocol and context broker input.

In other words, RTSoft platforms connector allows edge tier IoT devices to be represented in a FIWARE platform as NGSI entities in a context broker. This means that later on user can query or subscribe to changes of device parameters status by querying or subscribing to the corresponding NGSI entity attributes at the context broker. Additionally, user is able to trigger commands to actuation devices by updating specific command-related attributes in the associated NGSI entities representation at the context broker. This way, all hardware interactions with IoT devices from enterprise tier applications can be handled by the context broker, using a homogeneous NGSI interface.

To perform a translation task, mentioned above, it is required to have some kind of data mapping (or data model in Orion terminology). Initially, Orion context broker is intended to be used with such domains as Smart Cities, Smart Agrifood, Smart Energy, Smart Water etc. It does mean, corresponded data models are available out-of-the-box. However, it is quite possible to support any other domains by creating any custom data model, following NGSI-LD specification (https://json-ld.org/) see sample description of geo point object below:

{

"type": "Point",

"coordinates": [ -3.691944, 40.418889 ]

}

Data model creation is quite easy task to do, and could be done either by RTSoft, as solution provider, or by solution end-customer itself.

4.4. RITMS deployment and security aspects

RITMS solution is initially designed to provide end-users with a wide spectrum of various options to deploy it and support required data protection level. The key role here is a well thought base components selection (Enmasse.IO and FIWARE Orion). They have many deployment and security abilities built-in, so, RITMS solution allows you to utilize them the proper way.

Speaking about deployment, there are several options how to do it, exactly (see Pic.4.4.1 below):

  • All components are installed in the separate public cloud environments (MS Azure, AWS, Google etc);

  • Enmasse.IO and RTSoft platforms connector are installed in on-premise cloud infrastructure, and Orion FIWARE is installed in public one (or vice versa, in this case FIWARE Lab is a good choice for Orion);

  • All components are installed in separate on-premise cloud environments;

  • All components are installed in one single cloud environment.

Such deployment flexibility allows RITMS users to select one that better satisfies the particular project requirements. The only small technical limitation here is a necessity of RTSoft platform connector to be installed in the same environment where Enmasse.IO is running.

Pic.4.4.1

Enmasse.IO platform is distributed with own security sub-system onboard. It allows RITMS users to utilize such ability to provide required data protection for a whole solution – this is a preferable security providing option there. It allows to manage authorization / authentication processes “in one place” for all data flows which should be protected. Also, using collaboration with Enmasse.IO device registry component, built-in security sub-system usage approach allows automatic device provisioning process support.

Pic.4.4.2

However, RITMS is not limited by data protection option, mentioned above, only. For some specific use cases RITMS user can also utilize some 3rd party security sub-system instead of built-in one, or use separate security components for Enmasse.IO and Orion. The final security option selection depends of particular project requirements as well as deployment approach.

5. RITMS in practice – distributed power quality management system

Vivid practical example of RTSoft IoT Management Service usage is a project, which has been done by RTSoft last year. The customer of this project is a huge Russian government company, which operates in power industry. The main goal of the project was to develop, set in operation and support a complex distributed system for power quality monitoring and management (PQMS). RTSoft has successfully done this work, and now PQMS is installed and actively used in Russian Far East power region (~100 power objects, each of them is equipped by roundly 5-7 metering points (devices)).

Actually, this project is very close to a canonical IoT one, and has all required tiers: edge, platform and enterprise ones. RTSoft responsibility was to create first two tiers (including metering and gateways hardware distribution), enterprise tier was up to another company development.

High level scheme of PQMS solution is provided on the picture 5.1 below.


Pic.5.1

Parameters group

Number of signals

Average power quality values for 1 min and 30 min periods, including phases and harmonics

5097

Flicker

12

Statistics (1 day), including phases and harmonics

1712

Statistics (7 days), including phases and harmonics

1712

Sags and swells

56

TOTAL

8589

Table 5.2

Edge layer consists of a set of power quality metering devices, which are installed on power objects. Each of the devices performs required measurements – see parameters list above (table 5.2). The total amount of signals, which circulates inside the system, is approximately 6 000 000.

Each device PQ measurements data flow is transmitted to the power object PQMS gateway, using IEC65860-5-104 data exchange protocol, which is well known in industrial automation. Special software module (arranged as Eclipse Kura add-on) is responsible for translation field IEC65860-5-104 protocol to MQTT IoT data flow. Such flow is an input for solution’s platform layer, which is developed using RITMS.

According to RITMS internal rules, Eclipse Kura data flow is transformed to the set of domain-driven context objects, such as “Power region”, “Power object”, “Metering point”, “Group of PQ parameters”, “Signal value” etc. FIWARE Orion context broker is initially designed mostly for “Smart city” domain purposes, and corresponded data model is provided out-of-the-box. However, it is not a big deal to follow internal Orion rules and create model for another domain, which has also been done by RTSoft for PQMS project.

PQMS is an internal corporate system for a big company, so it was decided to deploy platform tier (which is exactly RITMS) according to deployment option D (single instance, see Pic.4.4.1 above).

As it was mentioned above, enterprise tier applications (on top of platform tier) for PQMS system were a subject of another company software development work. It means, these developers should collaborate in practice exactly with RTSoft’ RITMS solution output, and be able to estimate all benefits of NGSI context-based information representation. They did it – the feedback was very positive, they emphasized a high usability level of provided by FIWARE Orion broker context-based NGSI protocol usage, as compared with AMQP one, in case of pure Enmasse.IO output collaboration.

6. RITMS benefits summary

Taking into account all hereinabove, complex reliable IoT solutions usage for a practical IoT domain projects development is quite evident. It allows engineering teams to save development efforts dramatically, through switching from usual “writing code” processes to mostly “setup and configuration” one.

On the other hand, it is really effective to create such complex solutions basing on available set of the open-source products. Why – it is also evident – the development companies could save a lot of money, be able to modify the code according to their particular needs, and rely on these products long-term support and up growth (assuming IoT open-source communities are usually stable and growing).

RITMS solution (see Pic.6.1 below as its role illustration) is created exactly according to the considerations above and intended to cover IoT project platform tier. It accumulates all general benefits, listed there, and, due to well-considered base powerful components selection, adds new ones:

  • Close integration with one of the best edge tier frameworks (Eclipse Kura), which provides almost seamless integration of the edge and platform IoT tiers;

  • Ability to create enterprise tier application quickly and easily, due to object-based format of RITMS output data flows;

  • Low efforts to set up RITMS solution for the new domain area (no need to write user own code, just create a configurations). No participation of RTSoft engineers is required to tune the solution up;

  • Great scalability and flexibility, because of close integration with specially designed to achieve such goals Kubernetes platform;

  • Various built-in features of RITMS-based platform tier deployment (both for public, hybrid and on-premise environments). Users can choose one, which satisfies particular needs better;

  • Various built-in security scenarios. Users can choose one, which better satisfies the requirements;

  • Ability to use already existing hardware facilities, which are provided by FIWARE Consortium.


Pic.6.1

All the benefits, listed above, are proven in practice, during the real RTSoft IoT projects development processes (see chapter 5 above, as example). It allows us to recommend RITMS as ready-to-use, powerful independent tool to facilitate the implementation of such kind of projects both for our partners and anybody, who are interested in it.