Practice 11 - IoT device integration with cloud services
In this lab, you will learn how to set up Cloud service sof IoT device and data management, We will create an IoT Hub service in Azure, create a simple Python-based IoT device software and integrate it with Azure IoT hub.
References
- Python guide for Azure IoT Hub: https://learn.microsoft.com/en-us/azure/iot-hub/device-management-python
- Creating IoT devices: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-create-through-portal#register-a-new-device-in-the-iot-hub
- Provisioning devices: https://learn.microsoft.com/en-us/azure/iot-dps/how-to-legacy-device-symm-key?tabs=linux&pivots=programming-language-python
- Python examples: https://github.com/Azure/azure-iot-sdk-python/tree/main/samples
- Python AMQP examples: https://github.com/Azure/azure-uamqp-python/tree/main/samples
PS! Code examples from these references have been used as source when designing this lab exercises
Exercise 11.1 Creating an IoT Hub service in Azure
We will configure the Azure IoT hub and register a new device record inside it.
Create a new IoT hub Service:
- Search for IoT Hub service
- Region: North EU
- Set a unique name which contains your last name
- Tier: Basic
Create a new IoT device:
- Open the Devices page under IoT Hub
- Add device
- Specify a device ID, which contains your last name
- Keep Authentication type: Symmetric key
Because we are using the free Basic tier, some of the functionality is disabled:
- We can not update the software running on the device from the cloud
- We can not send operations from the cloud to the device.
In the next task, we will create simple Python software for this new IoT device.
Exercise 11.2 Creating a new Python-based IoT device
- Create a new Python project (e.g. in Pycharm)
- Create a new method (or a module) named
generate_sas_token
- Take the method implementation from this guide: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-dev-guide-sas?tabs=python
- We are going to use AMQP protocol, but instead of the pika Python client (which we used for RabbitMQ), we are going to use the uamqp Python library
- Import it using the code:
import uamqp
- Also import libraries:
- uuid, urllib, json, os
- Import it using the code:
Let's set up some Python variables:
iot_hub_name
- value should be the name of your IoTHubhostname
- value should be the hostname of your IoTHub- e.g.:
iothubjako.azure-devices.net
- e.g.:
device_id
- value should be the ID of the IoT device you created inside your hubusername
- value should be in the format of:{device_id}@sas.{iot_hub_name}
- e.g.:
pelledevice@sas.iothubjako
- e.g.:
- Read access key value from environment variables
- access_key = os.getenv('ACCESS_KEY')
- Set up ACCESS_KEY variable in environment variables
- In Pycharm, you can configure it in run configuration - when executing the script. Under environment variables.
- You can copy the ACCESS_KEY the value from the device configuration (Both primary and secondary keys are suitable) inside the Azure IoT Hub.
- Let's now generate a sas token using the previously created
generate_sas_token
method:sas_token = generate_sas_token('{hostname}/devices/{device_id}'.format( hostname=hostname, device_id=device_id), access_key, None)
- Define the message path to be
'/devices/{device_id}/messages/events'
inside the broker:operation = '/devices/{device_id}/messages/events'.format( device_id=device_id)
- We will send messages there. This is the default message path for incoming telemetry messages from an IoT device.
- Define the full AMQP path:
uri = 'amqps://{}:{}@{}{}'.format(urllib.parse.quote_plus(username), urllib.parse.quote_plus(sas_token), hostname, operation)
- It includes the username, authentication token, hostname of the service, and the message path.
- @ character in username is quoted using
urllib.parse.quote_plus
method to avoid breaking the separation of username, password, and URL in the amqp URL. - We also use
urllib.parse.quote_plus
to escape any special characters inside the generated token string.
- @ character in username is quoted using
- It includes the username, authentication token, hostname of the service, and the message path.
Now we can initiate the AMQP client and define the message to be sent from our IoT device.
- Initiate the uamqp sender client:
send_client = uamqp.SendClient(uri, debug=True)
- Define message:
- Create a new MessageProperties object for message metadata:
msg_props = uamqp.message.MessageProperties()
- generate message id:
msg_props.message_id = str(uuid.uuid4())
- create a new JSON message:
message_json = {'temperature': 22 }
- Convert the message to a string object:
msg_data = json.dumps(message_json)
- Define uamqp Message object:
message = uamqp.Message(msg_data, properties=msg_props)
- Create a new MessageProperties object for message metadata:
Let's now send the message
send_client.send_message(message)
- You can check/print the content of the results object to see if the sending of the message succeeded.
None
is ok.- If the value is equal to
uamqp.constants.MessageState.SendFailed
variable value then sending failed.
You can verify from the IoT hub page whether the messages from the device have arrived.
- Check the "Device to cloud messages" statistics diagram on the Overview page of your IoTHub service. You should see at least one message arrival being visualized there.
Deliverable: Take a similar screenshot of a situation where you see that telemetry messages have arrived in IoT hub. Also, make sure the name of the IoT hub is visible from the screenshot (larger screenshot than the previous sample here)
Exercise 11.3 Getting access to the Device data
Let's create a Python script for listening to messages that our IoT device sends to the Event Hub
- Create a new Python program/script in your project
- Check this tutorial page: https://pypi.org/project/azure-eventhub/
- And use the example code from the "Consume events from an Event Hub asynchronously" section as a basis for your Python script for listening to messages sent to IoT hub from the IoT device
- Check this tutorial page: https://pypi.org/project/azure-eventhub/
- Install the
azure-eventhub
python package in your Python project - Configure the following input variables for the script
- eventhub_name - This is NOT IoT Hub name. You will find the event hub name from the "Built-in endpoints" page under "Event Hub-compatible name" field.
- Consumer group - You will find the name of the consumer group under the "Built-in endpoints" page in IoT Hub, under Consumer Groups. Do NOT remove the "$" character.
- Connection String - You will find under
Built-in endpoints -> Event Hub compatible endpoint
- Change the drop-down selection from iothubowner to service
- NB! Do not copy the connections trying into code directly. Set its value using the Environment variable and use code like
CONNECTION_STR = os.getenv('CONNECTION_STR')
to access its value.
- '* Otherwise, there is a significant risk that you will share the secret key when you submit the code.
- Remove the parameter
starting_position
from theclient.receive
method call to receive only new messages. - Let's also print out the message content:
- Add the following lines into the
on_event
method, which is executed on every arrived message:
- Add the following lines into the
print("Telemetry received: ", event.body_as_str()) print("Properties (set by device): ", event.properties) print("System properties (set by IoT Hub): ", event.system_properties)
Test that you are able to access the IoT data.
- Start the message listener process
- Use the IoT device to send data
Deliverable: Make a screenshot of a situation where you have two windows next to each other:
- First one should show the running IoT device
- Second one should show the data arriving in the message listener output
Exercise 11.4 Update what data is sent by the IoT device
- Update the code of the IoT device to generate IoT data similar to the last Practice session, where we read in Puhatu IoT data CSV files and sent one JSON message every 10 seconds.
- Use a similar approach to prepare the content of the JSON message.
- NB! DO NOT send messages too frequently! And do not leave the code running for an extended period. Otherwise, there is a risk that you will use up all the daily free quota of your IoT hub.
Exercise 11.5 Routing IoT data to other Cloud services
IoT hub mainly works as an integration platform for IoT devices and their arriving data. Data itself should be stored and analyzed by other cloud services. In the Hub, we can set up different routing rules to route data to other cloud services. In this task, we will create a routing, which stores the arriving IoT data in Azure Storage account.
- Make sure the resource group you are using (Inside the one you created the IoT hub) contains a storage account Azure service.
- If not, create one.
- Create a new storage container inside the storage account
- Name it:
iot-data
- Name it:
- In the IoT Hub, create a binding to store messages in some other azure service
- Open the Message routing page
- Add a new routing named telemetry-to-storage
- Add a new Endpoint of Type: Storage
- Endpoint name: choose freely yourself
- Pick a container: Choose the same
iot-data
container you created earlier - File name format: Can keep default
- This defines which sub folders will be created for storing the message files
- Encoding: Change this to JSON, so it is easier for us to read the content of the files
- Authentication type: keep Key-based
NB! If the platform now offers you to restore the original message routing, do so.
- Test the routing works by creating a new message from IoT device and checking the content of the
iot-data
container.
Deliverable: Make a screenshot of a situation where see objects created inside the iot-data
container
Exercise 11.6 Automate the IoT device registration
Let's now create a new version of our IoT device, which uses Azure SDK and is able to automatically join into the IoT Hub - without having to create its ID and credentials manually. We will use this to create multiple IoT devices.
- Create a new "Azure IoT Hub Device Provisioning Services" service
- make sure it uses the same Resource Group and Region as the IoT Hub
- Wait for the resource to be created
- Under the Linked IoT Hub settings page of the Provisioning Services
- Add a new link - to your existing IoT hub
- Under the Managed enrollments settings page
- Create a new enrollment group
- Attestation mechanism: Symmetric Key
- Freely choose a group name
- Under Iot Hubs, make sure Your IoT Hub is selected
- Create a new enrollment group
Let's generate a new key for an IoT device.
- This should be done outside the IoT device software for security considerations.
- Follow the
Derive a device key
step in this tutorial to generate new symmetric key for (each) new IoT device: https://learn.microsoft.com/en-us/azure/iot-dps/how-to-legacy-device-symm-key?tabs=linux&pivots=programming-language-ansi-c#create-a-symmetric-key-enrollment-group- REG_ID: it is the device ID that we want our device to have.
- For example: jakovits-device-01
- We would generate new ID's for every new device.
- KEY: Symmetric key from your Provisioning Service.
- You can find it under the ENROLLMENT Symmetric key you created inside the provisioning Service
- REG_ID: it is the device ID that we want our device to have.
Create a new Python program.
- Install
azure-iot-device
Python package inside your project - Import needed libraries:
import asyncio import uuid from azure.iot.device.aio import ProvisioningDeviceClient from azure.iot.device.aio import IoTHubDeviceClient from azure.iot.device import Message import json
- We are no longer using AMQP but rather use the Azure SDK.
- Define the needed variables:
provisioning_host = "global.azure-devices-provisioning.net"
id_scope
- You will find it under ID Scope on the Provisioning Service main Overview page- For example:
id_scope = "0ne009EE5E5"
(DO NOT USE this specific ID)
- For example:
request_key
- this is the symmetric key generated for this IoT device in the previous step.- NB! DO NOT write the key value into the script, but rather read it from the system environment variable or from the script command line parameters. This is needed for both security reasons and to be able to use the same script for multiple IoT devices later.
device_id
- this is the IoT device ID used in the previous step.- NB! DO NOT write the device ID value into the script, but rather read it from the system environment variable or from the script command line parameters. This is needed for both security reasons and to be able to use the same script for multiple IoT devices later.
- Create an asynchronous main() method
async def main():
- Call this method from the main program script:
if __name__ == "__main__": asyncio.run(main())
- Update the
main()
method in the following steps: - Let's create the provisioning client, which we use to register the device or check its registration status.
provisioning_device_client = ProvisioningDeviceClient.create_from_symmetric_key( provisioning_host=provisioning_host, registration_id=device_id, id_scope=id_scope, symmetric_key=request_key, )
- Let's initiate and wait for the registration to finish:
registration_result = await provisioning_device_client.register()
- We can check for the registration state by using:
if registration_result.status == "assigned": print("Registration succeeded")
- Inside the IF statement block, if the registration succeeded, let's create a new device client object:
device_client = IoTHubDeviceClient.create_from_symmetric_key( symmetric_key=request_key, hostname=registration_result.registration_state.assigned_hub, device_id=registration_result.registration_state.device_id, ) # Connect the client. await device_client.connect() print("Device connected successfully")
- Create the message JSON, also generating an ID to a new message using the
uuid
library:message_json = {'temp': 132} msg_data = json.dumps(message_json) message = Message(msg_data) message.message_id = uuid.uuid4()
- NB! Modify this part accordingly to your previous code to change what is the content of the message (e.g. Puhatu data)
- Send a message
await device_client.send_message(message)
Individual task
- Check that the messages arrive in the IoT Hub service
- Deploy and run multiple IoT devices with their own (different) ID's and keys.
Deliverable: Make a screenshot of a situation where it is visible that two different IoT devices are sending messages at the same time
- E.g., two terminals/windows side-by-side outputting results/logs
- Make sure the ID of the device is shown in the output. May have to add additional logging or printing statements.
Deliverables:
- Screenshots from exercises: 11.2, 11.3, 11.5, 11.6
- Code from exercises:
- 11.3 (Data listener)
- 11.4 (First IoT device after modifying what data is sent)
- 11.6 (Second, self-registering IoT device)
- DO NOT include any Python virtual environment folders/files (e.g.,
.env
folder content)
issues and possible solutions
- If you get an error
AttributeError: __aexit__
in Exercise 11.3- Try whether commenting out the following line helps:
async with client:
- to
#async with client:
- to
- Try whether commenting out the following line helps: