Internet of Things on AWS
The Internet of Things. It is a ‘word’ that has been around since the early 2000’s, but the concept has been around for much longer. IoT, as it is called, is a network of interconnected devices that can connect to each other and exchange data. Everywhere in our lives there is IoT,; manufacturing machine producing pharmaceutical products pushing metric data for monitoring, making the car industry more autonomous and logistics companies optimising their supply chain are just some example of how IoT has ushered them into a new age (Industrial IoT).
According to statistics, in 2021 28% of enterprises in Belgium used IoT devices and the number of interconnected devices will double to almost thirty billion by 2030. The need for IoT Increases every day and so does the maintenance.
With AWS IoT Core you can easily connect your devices to the cloud and leverage the use of AWS tools. Note that there is a difference between AWS IoT and AWS IoT Core; AWS IoT is the umbrella term that encompasses the entire suite of IoT-related services on AWS, including AWS IoT Core. AWS IoT Core, on the other hand, is a specific service within AWS IoT that focuses on the core connectivity and communication aspects of IoT devices with AWS IoT Core.
With AWS IoT Core, the focus is how to create an AWS IoT device (or ‘Thing’), configure MQTT messages, set up AWS IoT Jobs, … and that will also be our focus in this blog!
Things and MQTT messages
With this basic understanding of AWS IoT, we can start a small EC2 instance and use this as a IoT device to explore the many functionalities. So how do we make external devices interact with our AWS services. Is there a way to define this device in our AWS environment? There is and they are called ‘Things’. In the context of AWS IoT, a “thing” represents a physical device or an entity that interacts with the AWS IoT service. In the Internet of Things (IoT) principle, devices (things) are connected to the internet and can send and receive data. AWS IoT provides a framework for securely connecting and managing these devices, and a “thing” is a fundamental concept within this framework. For this demonstration we will be creating an EC2 instance which we will then define as a ‘Thing‘. Note that not only EC2 instances can be ‘Things’; in theory, every ‘smart’ device can be added to the interconnected network that is IoT.
EC2 is the service in AWS that allows for the creation of instances, or computers, without having to provision the OS, storage and all that comes into play like with physical computers. On these cloud computers we can run whatever application, software and so on we wish.
For this demo we can keep it simple with a t2.micro EC2 instance, which is the lowest cost machine in terms of storage, computing power and OS:
Keep in mind if you want to remotely access your EC2 instance, you should define a key-pair, one will be saved locally on your machine so you can SSH into your EC2 instance. Also allow for SSH traffic to your instance (default is 0.0.0.0/0 which may be a security issue).
With the EC2 instance created, we can now add this device to AWS IoT Core. Since it is only one device, we are adding for demo purposes, let us go ahead and follow the steps in AWS IoT to create the ‘Thing’. Some of the steps are quite simple, but one is particularly important:
The AWS IoT Device SDK is a set of software libraries and tools to help developers build applications for IoT devices that interact with AWS IoT services, like creating resources, buckets, storing metrics data and much more.
Before we can demonstrate one of the IoT Core functionalities, we must finish setting up our ‘Thing’. Finishing this set up will result in a connection tool kit zip file. This file contains the necessary public and private keys and certificates to allow for secure communication between our EC2 instance and AWS IoT Core. Let us move the zip file from our local machine to the EC2 instance using our local command line:
To be able to do this, you need the key-pair created when creating the EC2 instance, the public IP address or DNS name of your EC2 instance and our zip file that will connect our EC2 instance with AWS IoT Core. Before we can run the start.sh script in the zip file, we must install the following packages:
sudo yum install git -y && sudo yum install pip -y
We can now finish the walkthrough of adding our EC2 instance as a ‘Thing’ and run the start.sh script. We can immediately see the output in our command line appearing, as a sample python script creates a MQTT connection to AWS IoT Core. But what is MQTT, or Message Queuing Telemetry Transport?
It is the most widely used messaging protocol that allows for communication between devices and applications. MQTT relies on the Pub/Sub logic and in our case, messages are sent from our IoT device to AWS IoT Core using the sample python script, pubsub.py:
Using the pubsub.py file one can simply publish and subscribe to messages arriving in defined topics. With communication going from our ‘Thing’ to AWS IoT Core, we can also configure communication to go from AWS IoT Core to our ‘Thing’. For this communication to flow from AWS IoT Core to our IoT device, we must make a small tweak to the pubsub.py by supplying our python script with the following parameters:
python3 aws-iot-device-sdk-python-v2/samples/pubsub.py –endpoint a3ce53xe8jxdjn-ats.iot.eu-west-3.amazonaws.com –ca_file root-CA.crt –cert iot-testing-thing.cert.pem –key iot-testing-thing.private.key –client_id basicPubSub –topic sdk/test/python –message ” –count 2
By passing an empty string to our –message parameter, we listen to incoming messages from our endpoint (the –count parameter defines the number of messages can be received before the MQTT connection is closed).
Managing your IoT devices using AWS IoT Core Jobs
When you have thousands of IoT devices, it could become a hassle to have to manage them all separately, updating the software, installing files and more. With IoT Core Jobs, defining and running these tedious tasks can be largely handled by scheduled jobs.
At the heart of these Jobs are Job documents. These are JSON files that contain the logic for the job. You can define your own, but AWS has a select catalog of useful jobs to choose from: downloading files, rebooting your IoT device and running a command are just some of them. For generic tasks, these AWS managed templates are perfect, but for other use cases that are more business specific, I recommend a more custom way of templating.
Let us say we have a fleet of over a 1000 IoT devices, what now? First off, defining jobs can be either on ‘Thing’ level or on group level. So, there is a solution at hand if you have many devices for the same purpose; you can just define a ‘Thing’ group and apply the job to said group. But what if I want to add a ‘Thing’ later, will the job also run on that? With both a ‘Snapshot’ and ‘Continuous’ run type, the job can be defined to run on only the current ‘Things’ in the group or all future ‘Things’ that are added in the group. By adding the scheduling configuration layer to the jobs, you can even more tweak the defined jobs by scheduling what time to start the job run, when to end the job run and what time you schedule maintenance. They do kind of resemble cron jobs…
Working with IoT devices in AWS IoT Core can be quite easy if you are just looking to manage your devices remotely and occasionally update the firmware. It was fairly simple to set up a EC2 instance and to define it as a ‘Thing’. Also, by implementing simple functionality from the IoT Device SDK, a lot of business processes can be managed just with the basic level of IoT Core. But with the AWS IoT Device SDK there’s so much more possible beside basic MQTT Pub/Sub and job management. Imagine the power of the Device SDK and AWS Core when working with a fleet of thousands of IoT devices, managing all incoming data (into S3 buckets), building your data lake (using Glue) or data warehouse (Redshift) and building dashboards to visualise all the IoT data (QuickSight). IoT Core could be the first building block in a AWS centric data solution.
Beside the mentioned functionality which was only limited, but not exclusive to the Device SDK, there are some that unmentioned ones such as ‘Thing’ shadows, which is a IoT service that allows you to interact with a ‘Thing’ in your network, even when the ‘Thing’ is not in an active state. It provides a way to interact with said ‘Thing’ to manage the current and desired state of the device; it literally ‘shadows’ the device. Another IoT Core service that deserves some mentioning is IoT Greengrass, which allows for AWS services and functionality at the edge. What this means in layman terms is it extends the AWS cloud functionality to process data, run computations, device management and more on the devices themselves, which brings it much closer to the data. This can be especially beneficial when you have time critical processes, managing offline operations on the data,…
As always with AWS services, the entry level of their tools is quick and easy to grasp, but real value comes from deep understanding and tinkering with the available tools.
I am Ilias Bukraa and I love data engineering with a background in bioinformatics and passion in code, I started working at Aivix with the goal and ambition to use this passion and previous data engineering experience on cloud projects. At Aivix I received opportunities to obtain certification in both Databricks and AWS and my journey of exploring the data engineering world continues.