Practice 5 - Working with Cloud databases (Azure blob storage and Cosmos DB)

In this lab, we will use Azure Cloud databases and take a look at the Azure Blob Storage and Cosmos DB database services.

Azure Blob Storage is Microsoft's object storage solution for the cloud. Blob Storage is optimized for storing massive amounts of unstructured data. Some examples of using Blob Storage are used for:

• Serving images or documents directly to a browser.
• Storing files for distributed access.
• Streaming video and audio.
• Writing to log files.
• Storing data for backup and restore disaster recovery, and archiving.
• Storing data for analysis by an on-premises or Azure-hosted service.

You will use the existing account in Azure Cloud and set up and modify the message board application to store the receive and store images in Azure Blob Storage services. You will also use the Azure Cosmos DB NoSQL service as the database to store the messages. In addition, You will also test the application locally and then deploy it on Azure App Services.

Here is the flow of tasks:

In this practice session, we will re-use the message board application we worked with in the previous labs). We will extend the message board application by using different Azure database services:

1. Update the home.html with an option to upload images along with text messages and display the images on the web page along with messages.
2. Store the images in the Azure Blob Storage service.
3. Use the Azure COSMOS DB (No SQL database) to store the message data as JSON documents.
4. Finally, create and deploy a message board application to 'Azure App Services.

Additional materials, tutorials, and references

• Azure Cosmos DB - https://learn.microsoft.com/en-us/azure/cosmos-db/
• Azure Blob Storage - https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction
• Azure Blob Storage Python SDK - https://learn.microsoft.com/en-us/azure/storage/common/storage-samples-python
• Azure Cosmos DB SDK - https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/sdk-python

Exercise 5.1. Introduction to application and setting up of development machine

In this task, you will update the message board application also to support uploading images.

Set up the development environment

• PS! You can use your own computer or VM in OpenStack as a development machine. The guide assumes that you use OpenStack VM.
• Create a VM in OpenStack VM (Use ubuntu22.04 ).
• Install the Python virtual environment library, create a Python virtual environment, and activate it as you did in Exercise 1.4 in Practice Session 1.
• You should clone the following GitLab repository: https://gitlab.cs.ut.ee/cloudcomputing/practice5
  • It contains a Python Flask messageboard code similar to the one we used in the previous lab when we worked with Azure Functions. But it uses normal Python handleMessage and htmlForm methods instead of FaaS functions.

Now, modify the application accordingly to the following instructions:

• Update the home.html template so your name is visible on the web page.
• Update the home.html template so users can upload an image with their message.
  • Add one more line inside the HTML <form> element for creating a file upload form input. It must be after <form> and before Submit lines:
    • <input type="file" name="file"><br>
  • Add enctype=multipart/form-data inside the <form HTML element. This is required to be able to upload files through an HTML form.
    • It should look something like:
      • <form action="/handle_message" method="post" enctype=multipart/form-data >
  • Add <img src="{{ m.img_path }}" width="500" > at the end of the message line so that the image will also be displayed after the message.
    • The line should now look something like this:

<li>
  "{{m.content}}" <small>Posted on {{m.timestamp}} </small> <img src="{{ m.img_path }}" width="500">
</li>

• Create static/images directory (inside the root Flask project directory) to store the image files.
  • Flask automatically makes files inside the static folder publicly available.
• Update app.py to implement the logic of storing uploaded image files on disk and storing their path with the message content.
  • Create a new variable that defines where we store image:
    • UPLOAD_FOLDER ='./static/images'
  • Modify the handleMessage() method to store uploaded image into a file inside the static/images folder
    • First, let's set the image path to an empty string:
      • img_path = ""
    • Then, lets check that file value was provided inside the HTML form:
      • if('file' in request.files and request.files['file']):
    • Inside the If block, add code to fetch the file object, specify where to save it, and store it as a file:

image = request.files['file']
img_path = os.path.join(UPLOAD_FOLDER, image.filename)
image.save(img_path)

• Then, modify the append_message_to_file method call to also include an argument for the new image path:
  • append_message_to_file(img_path, new_message)

• Next, modify the def append_message_to_file() function:
  • Add one more argument img_path to the input of the function:
    • def append_message_to_file(img_path, content):
  • Add the img_path key into the message JSON for storing the location of the file to save the location of the image that was uploaded together with this specific message:

    •     new_message = {
              'content': content,
              'img_path': img_path,
              'timestamp': datetime.now().isoformat(" ", "seconds")
          }
• The sample code home.html looks like this
• The sample code of app.py look like this

After these updates, the application should work something like this:

• Test the application locally similarly to the first practical (IaaS) using the flask run --host=0.0.0.0 command.

Exercise 5.2. Working with the Azure Blob Storage

In this task, we will work with the Azure Blob storage. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn't adhere to a particular data model or definition, such as text or binary data.

• Let's install and configure Azure Command Line Interface (CLI) inside the VM:
  • Install Azure CLI curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
  • Log in to your Azure account using the command az login.
• Let's create an Azure resource group, storage account, and containers to store the images.
  • We will use Azure CLI commands as per guidelines from this manual and follow accordingly in the subsequent tasks below.
  • PS! Remember to replace placeholder values in angle brackets with your own values. Should ignore the angle brackets and use the values, for example, <resource-group> should be replaced with lab5
  • If you have multiple Azure subscriptions, then you can stick to the default Student Subscription by running the command az account set --subscription "Azure for Students"

Task 5.2.1: Create a Resource Group, Storage Account, and Storage Container to store the blobs.

• Create a resource group az group create --name <resource-group> --location <location>
  • name lab5
  • location northeurope
• Create a storage account

  az storage account create \
      --name <storage-account> \
      --resource-group lab5 \
      --location northeurope \
      --sku Standard_ZRS \
      --encryption-services blob 

  • Replace <storage-account> with lab5<last_name>. Ex: lab5poojara
• Create a storage container az storage container create --account-name <storage-account> --name images

Task 5.2.2: Create, download, and list the blobs

• Let us download the sample image file, or you can use any other wget https://estonianworld.com/wp-content/uploads/2022/03/University-of-Tartu.-Photo-by-Andres-Tennus.-1536x864.jpg -O UT.jpg
• Upload to the blob

  az storage blob upload --account-name <storage-account> \
  --account-key <key2> \
  --container-name images \
  --file <path to file> \
  --name <blob name>
  

  • You can get the value of key2 by using the command az storage account keys list --account-name <storage-account> PS! Use the key2 value as the account-key.
  • --file could be ./UT.jpg (Pointing to the local file)and --name could be my_blob_UT.jpg (To be blob file name)
• List the blob using the command

az storage blob list --account-name <storage-account> \
--account-key <key2> \
--container-name images \
--output table

• Download the blob using the command

  az storage blob download --account-name <storage-account> \
  --account-key <key2> \
  --container-name images \
  --name <blob name> \
  --file <path-to-file>
  

  • Here, <path-to-file> is a path in your local directory where you want to store the blob. Ex: ./my_blob_image.jpg

Task 5.2.3: Working with blob permissions

Let's try to access the blobs and set the public access permissions.

• Get the URL of the blob.
  • You can log in to the Azure portal and under Storage Accounts --> <storage-account> -->Data Storage -> Containers --> images, Click on the my_blob_UT.jpg.
  • Copy the URL and open it in the browser.
  • You will see the error with ResourceNotFound. This indicates that the blob doesn't have public access.
• Let us change the access level with anonymous-read-access as shown in the figure or the guide here. This should be done for containers with the name: images
• First, enable "Allow Blob anonymous access" for the whole storage account - You can find this option under the storage account Configuration page:

• Then, enable "Anonymous access level: Blob" for the container images

• Again, open the URL in the browser, and now you should see the contents of the blob.
• You can also use the Azure CLI to set blob and container permissions. Please refer to the az command here

Exercise 5.3 Using Azure blobs in flask application

Here, we are going to modify the message board application to store the images uploaded by the end-user in Azure Blob storage using Python SDK.

• Make sure you are under the application directory and activated the python virtual environment.
• Add python SDK for azure blob storage azure-storage-blob==12.3.1 in requirements.txt.
• Set environment variables STORAGE_ACCOUNT=<storage-account>, CONN_KEY=<key1> as an environment variables.
  • In Linux can be set using export, for example, export STORAGE_ACCOUNT=<storage-account>
  • These variables are temporarily stored and need to be set again if you exit and open the terminal, So I would recommend noting down also in Notepad with varibale_name and value.
  • Get the account key (key1) of storage account using az storage account keys list --account-name <storage-account>. Copy the key1 part and set the environment variable CONN_KEY
• Now, let us update app.py to use azure blobs
  • Import the Azure blob storage library from azure.storage.blob import BlobServiceClient
  • Read environment variables
    • CONN_KEY= os.getenv('CONN_KEY')
    • storage account name storage_account = os.getenv('STORAGE_ACCOUNT')
    • Container name images_container = "images"
  • Create connection client blob_service_client = BlobServiceClient(account_url="https://"+storage_account+".blob.core.windows.net/",credential=CONN_KEY)
  • Now, write a function insert_blob in app.py to upload the image into Azure blob.
    • The function receives the input name as filename insert_blob(img_path)
    • Get the filename of the image filename = (img_path).split('/')[-1]
    • You can refer to this document for an example to upload an item to blob using the Azure blob python library.
      • Create a blob client using the local file name as the name for the blob blob_client = blob_service_client.get_blob_client(container=images_container, blob=filename)
      • Upload the image file to the blob

 
    with open(file=img_path, mode="rb") as data:
        blob_client.upload_blob(data,overwrite=True)

• Update handleMessage(): method
  • Now call the function insert_blob(img_path) just after the image.save(img_path) with the saved image path as the argument.
    • Do not remove the image.save(img_path) line. We still need the image file to be stored on a local disk.
  • Further, let's change how the append_message_to_file gets called in the handleMessage() method.
    • The img_path value in data.json should be the image Blob Storage URL instead of the local disk path.
    • For this, you can change the input parameters of append_message_to_file calling method to: append_message_to_file(blob_path, new_message)

 
blob_path = 'https://'+storage_account+'.blob.core.windows.net/'+images_container+'/'+image.filename 
append_message_to_file(blob_path,new_message)

• Install the Python packages we added to requirements.txt file
• Test the application by adding a few messages with images
• (Not mandatory) You can also add the code to handle exceptions when no images are uploaded by the user.
  • Can use something like: if('file' in request.files and request.files['file']):
  • PS! Make sure to allow posting a message without an image file.
• Deliverable: Take a screenshot of the terminal output by running the command to list the blobs (az storage blob list).

Exercise 5.4 Using COSMOS Database in flask application

Azure Cosmos DB is a fully managed NoSQL database service for modern app development. In this task, we will use COSMOS DB to store the messages in the database similar to the serverless functions we created in the previous lab. But in this practice session, we will use the COSMOS python SDK to interact with Azure cosmos DB.

• Create a COSMOS DB No SQL database similar to Exercise 4.2.
  • Choose: Azure Cosmos DB for NoSQL
  • NB! Keep Serverless as the Capacity mode (other types will start using up credit!)
    • Note - If you are asked to choose a Workload type, use "Development/testing".
  • Resource group should be lab5 as created in the previous exercise.
  • Account name: lab5cosmoslast_name (Ex: lab5cosmospoojara)
  • Database id: lab5messagesdb
  • Container id: lab5messages

We are modifying the message-board flask application to store the messages in the database instead of locally at data.json. Here, we will modify app.py to interact with cosmos db.

• Make sure that you are under the project directory in the development machine
• Add entry of azure-cosmos in requirements.txt
  • Don't forget to do pip install
• Now let us get the endpoint COSMOS_URL and Master key to access the COSMOS DB using python SDK, You can use Azure CLI for this and set the values as environment variables (COSMOS_URL and MasterKey)
  • Replace <resource-group> and <account-name> (COSMOS DB account name) in the command
    • Endpoint COSMOS_URL az cosmosdb show --resource-group <resource-group> --name <account-name> --query documentEndpoint --output tsv
    • MasterKey az cosmosdb keys list --resource-group <resource-group> --name <account-name> --query primaryMasterKey --output tsv
• Recommend to notedown COSMOS_URL, DATABASE_ID,CONTAINER_ID,MasterKey in notepad.
• Now let us modify the app.py
  • Import the cosmos lib import azure.cosmos.cosmos_client as cosmos_client
  • Declare COSMOS_URL, MasterKey
    • Values can be gathered for example COSMOS_URL= os.getenv('COSMOS_URL')
    • COSMOS_URL, MasterKey should also be defined/used as Enviorenment variables! (e.g. using export command in Linux)
  • Declare the variables
    • DATABASE_ID='lab5messagesdb'
    • CONTAINER_ID='lab5messages'
  • Create cosmos client cosmos_db_client = cosmos_client.CosmosClient(COSMOS_URL, {'masterKey': MasterKey} )
  • Database client for connection cosmos_db = cosmos_db_client.get_database_client(DATABASE_ID)
  • Container connection string container = cosmos_db.get_container_client(CONTAINER_ID)
  • Write a function insert_cosmos, as similar to append_message_to_file
    • Which takes content and img_path as input
    • Add one more element id in new_message json and value can be generated using uuid.uuid4(). PS! It should be string type.
      • 'id': str(uuid.uuid4()),
    • Insert an item into the database and also need to import the following library:
      • import azure.cosmos.exceptions as exceptions

    try:
        container.create_item(body=new_message)
    except exceptions.CosmosResourceExistsError:
        print("Resource already exists, didn't insert message.")

• Write a function to read the messages from the cosmos read_cosmos as similar to read_messages_from_file
  • Read the items from cosmos messages = list(container.read_all_items(max_item_count=10))
  • return the messages list as the output of the function

• Now, update the htmlForm() function of app.py
  • Replace the read_messages_from_file method call with read_cosmos method call
• Update the handleMessage() function of app.py
  • Replace the append_message_to_file method call with insert_cosmos method call
• Update the htmlForm() function of app.py
  • Update the render_template method call second argument to "data" instead of data["messages"]
• Test the application
• Test the application by inserting new messages with images.
  • Deliverable: Take a screenshot of the development machine terminal with the command output of "flask run". The screenshot should contain a few GET and POST operations handled by the flask application.

Exercise 5.5 Deploying application in Azure App Service

We are going to use Azure git-based deployment of a message-board application to Azure App Service. We will use Azure CLI to perform the tasks.

• Initially, we need to create an Azure Service Plan az appservice plan create --resource-group lab5 --name lab5plan --is-linux --sku F1
  • PS! If you encounter an error associated with Free plan limitations, you can utilize the free service plan established in the previous lab. Access it by navigating to the Azure portal- (Azure Service Plans).
• Now, create an Azure APP Service with git-enabled deployment so that we can push application code with git directly to Azure platform: az webapp create --resource-group lab5 --plan lab5plan --name <app-name> --runtime "PYTHON:3.9" --deployment-local-git
  • Replace <app-name> with lab5<last_name>app (Ex: lab5poojaraapp)
  • The output of the above command contains a URL like: https://None@<app-name>.scm.azurewebsites.net/<app-name>.git inside a larger JSON document.
    • We will use this URL later, after replacing None with an actual username we get from the following commands
• Let's now specify and configure the location of the remote Azure git repository for our local project. This is where we will push the local code.
  • Get the credentials (Username and Password) that can be used to push the code to Azure git

az webapp deployment list-publishing-credentials --name <app-name> --resource-group lab5 --query "{Username:publishingUserName, Password:publishingPassword}" --output table

• This command outputs the username and password that we will need to use to push in the code in the next step.
• Copy and save the username and password, you will need to use them several times later.

• Add the following URL as the GIT remote repository target: git remote add azure https://<app-name>.scm.azurewebsites.net/<app-name>.git, here URL should looks something like: https://lab5xx.scm.azurewebsites.net/lab5xx.git

NB! In the following steps, Azure remote git repository can be named either "main" or "master", if using one gives an error, try the other.

• Push the code to master branch git push azure master. It will ask for a username and password. Use the credentials obtained in the previous step (Should be able to use copy & paste as the password is long).
  • You will see the set of application deployment logs.
  • If you get an error that "master branch does not exist"
    • Set deployment branch to "main":
      • az webapp config appsettings set --name <app-name> --resource-group lab5 --settings DEPLOYMENT_BRANCH=main
    • Push to main branch: git push azure main
• After successful deployment, open the application using URL https://<app-name>.azurewebsites.net/. However, you will see the application with Error Message. You need to create the Application Settings (Environment Variables).
• Now, let's create the application settings, which include all the necessary environment variables for our application
  • Blob-related variables: STORAGE_ACCOUNT, CONN_KEY
  • Cosmos DB related variables: COSMOS_URL, MasterKey
  • The following command configures a new application setting inside Azure that includes all these four variables.
    • NB! This command assumes that you have all these 4 variables set up as environment variables in the command line shell!
    • az webapp config appsettings set --name <app-name> --resource-group lab5 --settings STORAGE_ACCOUNT=$STORAGE_ACCOUNT CONN_KEY=$CONN_KEY COSMOS_URL=$COSMOS_URL MasterKey=$MasterKey
• You need to modify app.py to slightly change the names of these variables inside the code. You should add the APPSETTING_ prefix to all four of them.
  • For example STORAGE_ACCOUNT=os.getenv('APPSETTING_STORAGE_ACCOUNT').
  • Do this for all the four variables accessed from environmental variables:
    • CONN_KEY, STORAGE_ACCOUNT, COSMOS_URL, MasterKey
• Add and commit the changes into git:
  • git add . && git commit -m "Modifed app.py for app settings"
• Deploy the code again git push azure master (or to main branch)
• For debugging, you can access the logs using your terminal az webapp log tail --name <app-name> --resource-group lab5
• Open your application in the browser.
• Deliverable: Take a screenshot of the web page of the application containing at least one message and an image (Your web address should be visible).

PS! You can go to the development web interface of your App service to get a nice overview of logs, parameters, and files:

• https://<APP_SERVICE_NAME>.scm.azurewebsites.net/

Bonus task

As a bonus task, your goal is to fix some of the issues that the completed web application has and allow users also to specify their name when posting messages:

(Show Bonus tasks)

Deliverables:

1. Screenshots from Exercise 5.3, 5.4 and 5.5
2. Application source code (Do not include any env .env or .venv folders)
3. Link to your Azure messageboard application deployed as Azure App Service
   • Keep the App Service running at least until you receive feedback.
4. Delete any OpenStack instances you have created

In case of issues, check these potential solutions to common issues:

• If you get an error about empty "main" branch or "master" branch not existing when deploying your code to Azure git:
  • Reconfigure what is the main branch name:
    • az webapp config appsettings set --name <app-name> --resource-group lab5 --settings DEPLOYMENT_BRANCH=main
  • And push to that branch then: git push azure main
    • If main branch does not exist, you can also create it (in remote azure repository) by using git commands.
• If images are not loading properly. Check if image files include spaces or special characters.
  • Try to upload images without those.
  • Or modify code to fix such image names.
• If you experience an error message "Sorry, we are currently experiencing high demand in this region ..." on service creation, then choose a different Region for example, Sweden Central
• If you get an error about ./static/images folder or ./static/images/somefile.jpg not existing
  • Make sure you created the static/images folder inside the Flask project root folder.
• If the application does not load.
  • If you have not used the application for a while, you are likely experiencing the Cold-Start issue, and it may take several minutes for Azure to provision your code again in the cloud.
• If you get an Error in Azure logs:

  • 2023-03-15T15:30:24.126415205Z if not 0 <= time_low < 1<<32L:
    2023-03-15T15:30:24.126427005Z ^
    2023-03-15T15:30:24.126432005Z SyntaxError: invalid syntax

  • You should not use uuid in requirements.txt file.
  • Rebuilding the Python virtual environment (.env folder) from scratch without custom uuid, removing uuid from requirements.txt and redeploying should solve it.
• If Flask is not able to access the Azure libraries
  • Double check that flask command is not running outside virtual environment.
  • If you installed flask on your own PC outside python virtual environment, it is possible it is using another (e.g., system env) environment instead
    • Possible solution would be to uninstall the flask library outside Python virtual environment.