DevOps: Automating Software Delivery and Operations - Courses

Practice session 6: Design of python flask microservices and APIs

Microservices is a variant of service-oriented architecture that consists of loosely coupled services composed in an application. It also enables the rapid, frequent and reliable delivery of large, complex applications. This lab aims to get acquainted with python flask microservices and create APIs using OpenAPI generator and accessing using Swagger. Further, you will deploy this set of services using docker-compose.

The following tools/libraries are used in this lab to design microservices.

OpenAPI : OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs, which allows both humans and computers to discover and understand the service's capabilities without access to source code documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with minimal implementation logic. More information about OpenAPI
Swagger : It's OAS specification to design the Restful APIs. Swagger allows you to describe the structure of your APIs so that machines can read them. The ability of APIs to describe their own structure is the root of all awesomeness in Swagger. We are using Swagger Editor to design the python APIs and validate using Swagger User interface More information about Swagger

Prerequisites

We are using IoT data of bog health monitoring system. Download the .csv file from here
It consists of nine days of data, recorded at different intervals in a day. There are three devices in the data set. The dataset consists of following readings from IoT devices

datetime: Timestamp of data when its recorded
batt: Battery level of the device
dev_id: Id of the device
snr: Signal to Noise Ratio
wat_Pressure_mH2o: Water Pressure
wat_Temp_float: Water Temperature
dist: Height of the bog
device_name: Device name

Exercise 1: Setting up of Swagger UI and Swagger Editor

The goal of this exercise is to get acquainted with Swagger components such as Swagger Editor and Swagger UI. Here, you will learn about YAML used to design the API documentation using OpenAPI specifications.

Create a Virtual Machine (VM)
- image : Ubuntu 20.04
- flavor: m3.tiny
- allow-all security group
- Assign floating IP
Login to your VM
Now you should be inside your $HOME directory.
Install Docker as per the steps mentioned in Practice session
Install Docker compose
- Download and save the executable file sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
- Change the mode of downloaded docker-compose file using chmod command: sudo chmod +x /usr/local/bin/docker-compose
- Verify the installation of docker-compose using docker-compose --version command.
Create a docker compose file docker-compose.yaml in your VM with following content:

version: '3.3'
services:
  swagger-ui:
    image: swaggerapi/swagger-ui
    container_name: swagger-ui
    ports:
      - "8001:8080"
    volumes:
      - ./swagger:/usr/share/nginx/html/swagger
    environment:
      API_URL: swagger/api.yaml
  swagger-editor:
    image: swaggerapi/swagger-editor
    container_name: swagger-editor
    ports:
      - "8002:8080"

Run the docker compose file docker-compose up -d (Make sure that the created docker-compose.yaml is present in the current path)

Now open any web browser available in your laptop/PC and visit http://VM_EXTERNAL_IP:8002
Now, you should be able to access the Swagger Editor, something similar to the below image:

In the browser, left side you should see the editor, with some YAML content. This is the place to write your Swagger configuration file. (Recall the Swagger/OpenAPI configuration file from the lecture)
Right side, you see the Swagger UI. Any modification to the YAML configuration will instantly reflect here. So, here also, you will instantly get errors if there is an error in the configuration editor.
- Get handy with tabs in the editor
  - File: This helps you to create new yaml, import existing yaml and other operations.
  - Edit: This helps you to convert json to yaml and other operations such as convert from OpenAPI versio 2 to version 3 document.
  - Generate Server: Once you finish designing the specification document, you can generate the template for your server-side code for various languages.
  - Generate Client: This helps to generate client-side code to invoke the APIs.
- By default you will see an example of Petstore server. So in the editor, you see the swagger configuration file of Petstore app.
  - You may get aquinted with this example and explore this application.
  - Go through API documentation of Pet Store application (right side of the swagger editor window).
- Try to understand following keywords/keys in the YAML configuration file (left side of the window)
  - swagger version
  - info
  - host Your API endpoint address
  - basePath Your API version with base location of your API.
  - Learn about namespaces, tags, definations from the YAML configuration document.(For information click here)
    Learn the namespaces and corresponding methods.
Screenshot 1-1: Take the screenshot of the Swagger editor web UI (should include the address bar of the browser)
Screenshot 1-2: Execute the docker ps command in the terminal and take the screenshot

Exercise 2: Working with Swagger Editor and creating a python-flask microservice

This exercise aims to write your (REST API for python-flask microservice) OpenAPI/swagger configuration documentation using Swagger Editor and generate the corresponding server-side code.

Open swagger editor by visiting http://VM_EXTERNAL_IP:8002.
Select File-->Clear Editor and than start writing your own configuration
- the first line should be the OpenAPI version. Lets use swagger: "2.0"
- info - The info section contains API information: title, description (optional), version, etc.

info:
  description: "This is documentation for Python Flask micro-service endpoints"
  version: "1.0.0"
  title: "Swagger IoT data"
  termsOfService: "http://swagger.io/terms/"
  contact:
    email: "xxx@ut.ee"
  license:
    name: "Apache 2.0"
    url: "http://www.apache.org/licenses/LICENSE-2.0.html"

host indicates your server address where your microservice will run. You can use VM_EXTERNAL_IP as the server address. It should look similar to host: "172.17.89.129:8081"
basePath indicates the endpont path. It can be basePath: "/v2"
schemes indicates either http or https invocation, if https than need to proived authourization mechanism. In this exercise, we will use http. Add the element http under schemes:. It should look like below:

schemes: 
  - http

paths are endpoints (resources), such as /users or /reports/summary/, that your API exposes, and operations are the HTTP methods used to manipulate these paths, such as GET, POST or DELETE. Here we add POST operations under /upload namespace. The tags indicates that this operation belongs to /upload namespace. An operation definition includes parameters, request body (if any), possible response status codes (such as 200 OK or 404 Not Found) and response contents. (For more information)

paths:
  /upload:
    post:
       tags:
       - "Upload Sensor Data"
       summary: Uploads a file.
       consumes:
         - multipart/form-data
       parameters:
         - in: formData
           name: upfile
           type: file
           description: The file to upload.
       responses:
        200:
          description: OK
        400:
          description: Bad request. User ID must be an integer and bigger than 0.
  /modify:
    put:
      tags:
      - "Modify Sensor Data"
      summary: "Updated Sensor name"
      operationId: "updateSensorData"
      consumes:
      - "application/json"
      - "application/xml"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - in: "body"
        name: "body"
        description: "device object that needs to be modified"
        required: true
        schema:
          $ref: "#/definitions/device"
      responses:
        "400":
          description: "Invalid ID supplied"

We continue path with adding /sensor_data namespace that consumes a dev_id as parameter and produces a json or string responses for different operations GET, DELETE.

 /sensor_data/{deviceId}:
    get:
      tags:
      - "Query Sensor Data"
      summary: "Find sensor data  by ID"
      description: "Returns a list of sensor data"
      operationId: "getSensorById"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - name: "deviceId"
        in: "path"
        description: "Device id to search in the data"
        required: true
        type: "integer"
        format: "int64"
      responses:
        "200":
          description: "successful operation"
          schema:
            $ref: "#/definitions/device"
        "400":
          description: "Invalid ID supplied"


    delete:
      tags:
      - "Modify Sensor Data"
      summary: "Delete sensor data"

      operationId: "deleteSensorData"
      produces:
      - "application/json"
      parameters:
      - name: "deviceId"
        in: "path"
        description: "Sensor Id that need to be updated"
        required: true
        type: "integer"
        format: "int64"
      responses:
        "200":
          description: "Successful operation"
        "400":
          description: "Invalid device id supplied"

like wise, we add the /minimum{sensor} and /maximum{sensor}, Over here, you will see a new key called definitions at the end. Try to search the purpose of this section.

/minimum/{sensor}:
    get:
      tags:
      - "Query Sensor Data"
      summary: "Find minimum sensor data  by ID"
      description: "Returns a list of sensor data"
      operationId: "getminimum"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - name: "sensor"
        in: "path"
        description: "Device id to search in the data"
        required: true
        type: "string"

      responses:
        "200":
          description: "Successfull with response containing minimum value"
          schema:
            $ref: "#/definitions/device"
        "400":
          description: "Invalid ID supplied"

  /maximum/{sensor}:
    get:
      tags:
      - "Query Sensor Data"
      summary: "Find maximum sensor data  by ID"
      description: "Returns a list of sensor data"
      operationId: "getmaximum"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - name: "sensor"
        in: "path"
        description: "Device id to search in the data"
        required: true
        type: "string"

      responses:
        "200":
          description: "Successfull with response containing maximum value"
          schema:
            $ref: "#/definitions/device"
        "400":
          description: "Invalid ID supplied"

definitions:
  device:
    type: "object"
    required:
    - "name"
    - "photoUrls"
    properties:
      dev_id:
        type: "integer"
        format: "int64"
      device_name:
        type: "string"
        example: "puhatu_c1"

The final configuration should look like below.

swagger: "2.0"
info:
  description: "This is documentation for Python Flask micro-service endpoints"
  version: "1.0.0"
  title: "Swagger IoT data"
  termsOfService: "http://swagger.io/terms/"
  contact:
    email: "poojara@ut.ee"
  license:
    name: "Apache 2.0"
    url: "http://www.apache.org/licenses/LICENSE-2.0.html"
host: "172.17.89.129:8081"
basePath: "/v2"

schemes:
- "http"
paths:
  /upload:
    post:
       tags:
       - "Upload Sensor Data"
       summary: Upload IoT data to server.
       consumes:
         - multipart/form-data
       produces:
         - application/json
       parameters:
         - in: formData
           name: upfile
           type: file
           description: The file to upload.
       responses:
        200:
          description: Successfully uploaded
        400:
          description: Bad request 
  /modify:
    put:
      tags:
      - "Modify Sensor Data"
      summary: "Updated Sensor name"
      operationId: "updateSensorData"
      consumes:
      - "application/json"
      - "application/xml"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - in: "body"
        name: "body"
        description: "Pet object that needs to be added to the store"
        required: true
        schema:
          $ref: "#/definitions/device"
      responses:
        "200":
          description: "Successful"
        "400":
          description: "Invalid Device"

  /sensor_data/{deviceId}:
    get:
      tags:
      - "Query Sensor Data"
      summary: "Find sensor data  by ID"
      description: "Returns a list of sensor data"
      operationId: "getSensorById"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - name: "deviceId"
        in: "path"
        description: "Device id to search in the data"
        required: true
        type: "integer"
        format: "int64"
      responses:
        "200":
          description: "Successful operation with json containing all the sensors with sensor id"
          schema:
            type: object
            properties:
              dev_id:
                type: integer
                description: The user ID.
              device_name:
                type: string
                description: The user name.
              snr:
                type: integer
        "400":
          description: "Invalid ID supplied"



    delete:
      tags:
      - "Modify Sensor Data"
      summary: "Delete sensor data"
      description: "This can only be done by the logged in user."
      operationId: "deleteSensorData"
      produces:
      - "application/json"
      parameters:
      - name: "deviceId"
        in: "path"
        description: "Sensor Id that need to be updated"
        required: true
        type: "integer"
        format: "int64"
      responses:
        "200":
          description: "Successful"
        "400":
          description: "Invalid device id supplied"

  /minimum/{sensor}:
    get:
      tags:
      - "Query Sensor Data"
      summary: "Find minimum sensor data  by ID"
      description: "Returns a list of sensor data"
      operationId: "getminimum"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - name: "sensor"
        in: "path"
        description: "Device id to search in the data"
        required: true
        type: "string"

      responses:
        "200":
          description: "Successfull with response containing minimum value"
          schema:
            $ref: "#/definitions/device"
        "400":
          description: "Invalid ID supplied"

  /maximum/{sensor}:
    get:
      tags:
      - "Query Sensor Data"
      summary: "Find maximum sensor data  by ID"
      description: "Returns a list of sensor data"
      operationId: "getmaximum"
      produces:
      - "application/xml"
      - "application/json"
      parameters:
      - name: "sensor"
        in: "path"
        description: "Device id to search in the data"
        required: true
        type: "string"

      responses:
        "200":
          description: "Successfull with response containing maximum value"
          schema:
            $ref: "#/definitions/device"
        "400":
          description: "Invalid ID supplied"

definitions:
  device:
    type: "object"
    required:
    - "name"
    - "photoUrls"
    properties:
      dev_id:
        type: "integer"
        format: "int64"
      device_name:
        type: "string"
        example: "puhatu_c1"

Check if you have any syntax errors in the document on the right side panel of the editor.
Click on Generate Server-->python-flask and it will download the server-side code for you in compressed format.
Extract the downloaded compressed file to your local machine (i.e. your laptop/PC) and you should have the following directory structure.

Now you need to modify swagger_server directory and DockerFile file to implement the functionalities mentioned in the configuration file.

You can find the configuration file inside swagger_server --> swagger --> swagger.yaml.
- Open the Dockerfile and replace the first line from FROM python:3-alpine to FROM amancevice/pandas:latest. This is because we are using pandas library in our further experiments with python 3.9.
- We need to modify upload_sensor_data_controller.py, modify_sesnor_data_controller.py and query_sensor_data_controller.py as shown in below figure.

Now let us modify the controller code upload_sensor_data_controller.py as below.

The logic/functionality is to receive the file(iot.csv) invoked using /upload POST REST interface and save in to the local directory in /tmp/iot.csv

import connexion
import six
from flask import jsonify
from swagger_server import util
import pandas as pd
import json

def upload_post(upfile):  # noqa: E501

    df = pd.read_csv(upfile)
    df.to_csv("/tmp/iot.csv")
    data = {"message": "File Successfully uploaded"}

    return data, 200

Like wise modify the query_sensor_data_controller.py with following code

import connexion
import six
from flask import jsonify

from swagger_server import util
import json
import pandas as pd


def get_sensor_by_id(deviceId):  # noqa: E501

    try:
        df = pd.read_csv("/tmp/iot.csv")
        if deviceId in df['dev_id'].values :
            df_deviceId = df.loc[df['dev_id'] == deviceId]
            df_resp = df_deviceId[['dev_id','device_name','snr']]
         #   df_resp = df_resp.head(1)
            data = df_resp.to_json(orient="records")

            status = 200
            #data = {"Success message": "Device deleted from the csv"}
        else:
            status = 400 
            data = {"Error message": "Invalid device"}
    except Exception as e:
            data = {"Error message": str(e)}
            status = 400 
    return json.loads(data),status

def getmaximum(sensor):  # noqa: E501
    """Find maximum sensor data  by ID

    Returns a list of sensor data # noqa: E501

    :param sensor: Device id to search in the data
    :type sensor: str

    :rtype: Device
    """
    return 'do some magic!'


def getminimum(sensor):  # noqa: E501
    """Find minimum sensor data  by ID

    Returns a list of sensor data # noqa: E501

    :param sensor: Device id to search in the data
    :type sensor: str

    :rtype: Device
    """
    return 'do some magic!'

Like wise modify the modify_sensor_data_controller.py with following code

import connexion
import six
from flask import jsonify

from swagger_server import util
import json
import pandas as pd
def delete_sensor_data(deviceId):  # noqa: E501
    """Delete sensor data

    This can only be done by the logged in user. # noqa: E501

    :param deviceId: Sensor Id that need to be updated
    :type deviceId: int

    :rtype: None
    """
    try:
        df = pd.read_csv("/tmp/iot.csv")
        if deviceId in df['dev_id'].values :
            indexNames = df[ df['dev_id'] == deviceId].index
            df.drop(indexNames , inplace=True)
            df.to_csv("/tmp/iot.csv")
            status = 200
            data = {"Success message": "Device deleted from the csv"}
        else:
            status = 400 
            data = {"Error message": "Invalid device"}
    except Exception as e:
            data = {"Error message": str(e)}
            status = 400
    return jsonify(data),status

def update_sensor_data(body):  # noqa: E501
    """Updated Sensor name

     # noqa: E501

    :param body: Pet object that needs to be added to the store
    :type body: dict | bytes

    :rtype: None
    """

          # noqa: E501
    try:
        body = connexion.request.get_json()

        dev_id = body['dev_id']
        device_name = body['device_name']

        df = pd.read_csv("/tmp/iot.csv")
        if dev_id in df['dev_id'].values :
            df.loc[df['dev_id'] == dev_id, 'device_name'] = device_name            
            df.to_csv("/tmp/iot_updated_modifed.csv")
            status = 200
            data = {"Success message": "CSV updated and saved as iot_updated_modifed.csv in /tmp"}
        else:
            status = 400 
            data = {"Error message": "Invalid device"}
    except Exception as e:
            data = {"Error message": str(e)}
            status = 400 
    return data, status

Now update the requirements.txt file with following code:

connexion[swagger-ui]
python_dateutil == 2.6.0
setuptools >= 21.0.0
flask_cors

Now lets update the swagger_server --> __main__.py file with following code:

#!/usr/bin/env python3

import connexion

from swagger_server import encoder
from flask_cors import CORS

def main():
    app = connexion.App(__name__, specification_dir='./swagger/')
    app.app.json_encoder = encoder.JSONEncoder
    CORS(app.app)
    app.add_api('swagger.yaml', arguments={'title': 'Swagger IoT data'})
    app.run(port=8080)


if __name__ == '__main__':
    main()

At this point, the whole swagger project is in your local machine.
Now, you need to push this to a new gitlab repository under your gitlab account.
From the previous Lab, you know how to create a gitlab repository locally and push to remote Gitlab server. Follow the same and push repository to remote Gitlab account.
Screenshot 2-1: Goto the newly created repository in web browser and take the screenshot of web page (should include the address bar of the browser)

Exercise 3: Deploying your python-flask microservices

Here, the goal is to learn about deployment of the python-flask microservice in docker environment. For this, you need to build the docker file to create an image for your microservice and use the same image to run.

Make sure you are in the terminal logged in to your VM
Clone the project from GitLab using git clone command
Change the directory to your project containing the Dockerfile
Use build command docker build -t <your_docker_hub_account>/flask-microservice .
Use the following compose file to run your microservice. So here you need to create a docker compose file with following content and issue docker-compose up -d command.

version: '3.3'
services:
  flask-microservice:
    image: <your_docker_hub_name>/flask-microservice
    container_name: my-flask-microservice
    ports:
      - "8081:8080"
    volumes:
      - /home/ubuntu/data:/tmp/
  grafana:
    image: shivupoojar/grafana
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - /home/ubuntu/data:/tmp/data

You should see the services running docker ps.
We test the microservice using Swagger User Interface
- Visit http://VM_EXTERNAL_IP:8001/.
  Load the API definition as shown in below sample.
  The Explorer textbox should look like below

Screenshot 3-1: take the screenshot of the web-page after the API definition is loaded. The address bar should be visible in the screenshot.
Make sure you have downloaded the iot data from here iot data
Now, let us test all the microservices endpoints of the python flask application using GET, POST,PUT and DELETE method. For this you need to press Try Out button at right side of the page on every method.
- POST : Upload the iot.csv file to python-flask server using POST method

Go to the http://VM_EXTERNAL_IP:3000 to see the uploaded data in grafana.
- Copy and save the grafana json template with iot.json in your local machine/laptop from here Dashboard Template
- Add a data source as CSV and path as /tmp/data
- Import the iot.json in grafana, Go to +-->Import-->Upload JSON file, select your iot.json
- You shoudl see the iot data as shown below

Screenshot 3-2: Take the screenshot of the response after executing POST method (address bar of the browser should be visible)

GET : Search for sensor data with sensorId

Screenshot 3-3: Take the screenshot of the response after executing GET method (address bar of the browser should be visible)

PUT : Update the sensor name from given sensor id

Screenshot 3-4: Take the screenshot of the response after executing PUT method (address bar of the browser should be visible)

DELETE : Delete the sensors data from the csv for a given sensor id

Screenshot 3-5: Take the screenshot of the response after executing DELETE method (address bar of the browser should be visible) and grafana dashboard.

Exercise 4: Home Work : Additional python-flask micro services

In this task, you need to update the query_sensor_data.py file by writing the code for getminimum abd getmaximum.

The sample controller code for minimum is given below, here values of sensor would be column headers in iot.csv

def getminimum(sensor):  # noqa: E501

    try:
        df = pd.read_csv("/tmp/iot.csv")
        df_sensor = df[[sensor,'dev_id','datetime','device_name']].min()
        data = df_sensor.to_json(orient="records")
        status = 200
    except Exception as e:
            data = {"Error message": str(e)}
            status = 400
    return json.loads(data),status

Like wise write the code getmaximum (Refer pandas guide for finding maximum column value in data frame)
Build the image and run the application.
Check the working of the methods using Swagger User Interface and please take the screenshots of the response of each method.

Screenshot 4-1: Take the screenshot of the response after executing GET method for /minimum/{sensor} namespace (address bar of the browser should be visible). Screenshot 4-2: Take the screenshot of the response after executing GET method for /maximum/{sensor}(address bar of the browser should be visible)

Deliverables

Upload the screenshot taken wherever mentioned
Pack the screenshots into a single zip file and upload them through the following submission form.

Deadline: 29th Oct 2021

6. Lab 6

Solutions for this task can no longer be submitted.

DevOps: Automating Software Delivery and Operations 2021/22 fall