Arvutiteaduse instituut
  1. Kursused
  2. 2024/25 kevad
  3. Pilvetehnoloogia (LTAT.06.008)
EN
Logi sisse

Pilvetehnoloogia 2024/25 kevad

  • Main
  • Lectures
  • Practicals
    • Plagiarism Policy
  • Results
  • Submit Homework

Practice 14 - Redesigning the Message-Board Application using open-source tools.

The goal of this lab is to migrate the Flask message board application that we have created from Azure cloud services to open-source solutions. In this lab, you will learn how to use two different open-source databases, such as JSON document database(MongoDB) and blob storage(MinIO). You will redesign the Message Board application to be deployed as a fully open-source solution in an OpenStack VM and make it run persistently across server restarts.

References

  • MongoDB: https://www.mongodb.com/
  • MinIO: https://min.io/
  • MinIO Python library. https://min.io/docs/minio/linux/developers/python/minio-py.html
  • Mongo Python library: https://pymongo.readthedocs.io/en/stable/tutorial.html#making-a-connection-with-mongoclient

Exercise 14.1 -Setting up an OpenStack VM

You will prepare a VM in OpenStack for deploying the application.

  • Create a VM in OpenStack Cloud with
    • Choose Image Ubuntu24.04
    • Enable Delete the Volume on Instance Delete
    • Flavor: g4.r4c2
    • make sure to activate Delete Volume on Instance Delete
    • Volume Size (GB): 20GB
    • Add Security Group lab14

Preparing the VM

  • SSH into the VM
  • Follow the Practice Session 2 - Docker lab to reconfigure the network for Docker and install Docker.

While developing the application, you can use the VM or your own computer to run the required software in Docker for testing. But you will have to set up the final version inside the Virtual machine.

Exercise 14.2 Working with Documents (JSON) based database: MongoDB

In this task, you will replace Azure COSMOS database with the MongoDB in your message board application. MongoDB stores data records as BSON documents. BSON is a binary representation of JSON documents.

Use the Message Board application code submitted in the deliverable of Practice Session 5 - Cloud Databases

  • Run MongoDB using docker container
    • Run in detached mode
    • Assign the name mongodb
    • Volume mount(-v) -v ${HOME}/mongo-data:/data/db
      • Create a directory mongo-data
    • Port forwarding 27017:27017
    • Enviorment variables to set username and password MONGO_INITDB_ROOT_USERNAME=sample-db-user, MONGO_INITDB_ROOT_PASSWORD=sample-password
      • Change the username and password values
    • Image mongo:latest
  • Modify app.py to insert JSON documents (Messages)
    • Delete azure cosmos imports and all the variables related to cosmos.
    • We will use PyMongo pip library for this task. Import pymongo and create a mongo client, for example here.
      • MongoDB client mongo_client = MongoClient("VM_IP", username = os.getenv('MONGO_INITDB_ROOT_USERNAME'), password = os.getenv('MONGO_INITDB_ROOT_PASSWORD'), port=27017)
        • Replace VM_IP with the IP of your VM or localhost if you are testing locally
      • Create a db db = mongo_client.flask_db.
      • Create a document collection messages=db.messages
    • Now, write a method to insert the documents similar to insert_cosmos().
      • Basically, you can rename the method.
      • Arguments remain the same.
      • You can use messages.insert_one(new_message) to insert document to MongoDB.
    • Similarly, modify the method read_cosmos to read the documents from MongoDB.
      • Get the latest 10 documents (messages). For this, you can use a similar to this.
        • Firstly, use find to get the records
        • Use sort to sort by id or sort([('timestamp', 1)])
        • Limit to 10 records using limit
        • Get them in a Python list using list(db.messages...).
  • Test the message board application.
    • Make sure you modified the following in app.py.
      • You need to comment out insert_blob(img_path) method invocation invocation in handle_message() and its declared variables and imports.
      • The blob_path arguement could be img_path for now in insert_mongo().
    • Call the method insert_mongo instead of insert_cosmos under handle_message().
    • Create a virtual environment and activate it.
    • Install pip requirements
    • Run the application and insert some messages.
  • You can connect to MongoDB shell
    • Exec to the mongodb container with bash
    • In mongodb shell, type mongosh
    • Write the queries to check your data. Some of query examples here.
      • List databases
      • Use your db
      • List the documents under messages collection
      • Query the list of documents with certain JOSN key equals. For example with username="Shiva".

Screenshot1: Take a screenshot of the terminal (mongodb shell) showing the output of the queries.

Exercise 14.3 Working with blob storage : MinIO

In this task, you will replace Azure Storage Blob service with the MinIO in your message board application. MinIO is a high-performance, S3 compatible object store. It is built for large scale data lake and database workloads.

  • Running a MinIO database (Single node) in a Docker container
    • Run in detached mode
    • Ports 9000:9000 and 9001:9001
    • Enviornment vars MINIO_ROOT_USER=username and MINIO_ROOT_PASSWORD=password
      • Change the username and password values
    • Volume mount ${HOME}/minio/data:/data
      • PS! make sure to create ${HOME}/minio/data directories
    • Image and starting command quay.io/minio/minio server /data --console-address ":9001"
  • You can access the MinIO web interface using http://VM_IP:9001
    • Login using your username and password.
  • Create a bucket named images
    • Administrator-->Buckets-->Create Bucket
  • Create Access Keys needed to work with MinIO storage using Python SDK.
    • Go to User-->Access Keys-->Create Access Keys
      • Set Expiry date of your own, Provide a name
      • Further click on Download for import to save the keys.
    • If you can not set Access keys from the user interface, investigate documentation for how to do it from the command line.
  • You can also access the MinIO through minio shell client mc
    • Exec to minio container and use mc client as below.
      • Provide your VM IP, accessKey, secretKey docker exec -it container_name mc alias set minio http://VM_IP:9000 accessKey secretKey
      • List the buckets mc ls minio. It should list your images bucket.
      • Set anonymous access policies to download the images from both the buckets using mc anonymous set download minio/images
  • Add minio to requirements.txt and install the pip requirements.
  • Now, let us modify app.py to store images in MinIO buckets. Here, We will use MinIO pip library and you can refer to guide here for information on Python SDK.
    • Delete all the imports of azure.storage and its declared variables.
    • Import MinIO library
    • Declare MinIO client as minio_client as described in the guide.
      • Replace "play.min.io" with VM_IP:9000
      • Replace access_key and secret_key values. PS! Don't put the values directly, they should be provided through environment variables.
      • Add one more argument secure=False
    • Modify the method insert_blob(img_path)
      • Keep the filename variable
      • Use minio_client.fput_object to upload the image to images bucket. (Check the previously linked guide for the complete statement)
        • bucket_name should be "images"
        • destination_file should be to filename
          • Should have only filename, for example sample.png without any path or so.
        • source_file should be img_path.
    • Updates in handle_message()
      • blob_path value should be 'http://VM_IP:9000/images/"+filename
      • Use blob_path as a arguement in insert_mongo().
  • Run the Flask application and insert some messages.
  • You should be able to see the images under images bucket in MinIO.

Screenshot2: Take a screenshot of the MinIO web page showing the list of images inside the bucket. (Make sure your IP is visible)

Task 14.4: Configuring services to stay running after VM restart

The goal of this task is to Containerize your application and make it run inside the OpenStack VM we set up in Exercise 14.1.

  1. Make sure MinIO and MongoDB are running inside the VM
  2. Configure MinIO and MongoDB Docker containers to be automatically restarted after machine restart:
    • docker update --restart=always container_name
      • Replace the container_name with the name of your MinIO and MongoDB containers, and run this command for both of them
  3. Containerize your application
    • Name the docker container image: lab14flaskapp
    • Make sure that:
      • Secrets are read from environment variables and not hardcoded into Python code and Dockerfile.
      • All necessary folders are created (e.g. ./images for keeping temporary files)
      • All necessary libraries are specified in the requirements.txt file
  4. Build and run your application container
    • Redirect traffic to a port (-p) that is accessible through the OpenStack Security group rules
    • Also, make sure you enable --restart=always
  5. Test that the application stays running after a restart
    • Restart the VM from the OpenStack web interface or from the command line
    • Check that you can still use the web interface of the application after a restart without any manual intervention.

Screenshot3: Take a screenshot of the Application web page. The IP of the VM must be visible.

Screenshot4: Take a screenshot of one of the images uploaded to the Application web page. The URL of the image must be visible (you can use "Inspect Element" in the browser to display the URL of the image).

Bonus Task: Creating a Docker compose file for the whole deployment

This is an individual task with minimal guidance. Your goal is to create a single docker-compose.yaml file where all the Docker containers are defined with all the necessary parameters.

  • MongoDB container
  • MinIO container
  • Python application Docker container
    • And setting all the necessary environmental variables.

(Show Bonus Instructions)

  • It is suggested that you put the authentication details into a separate .env file in the same folder, rather than write the values directly into the docker-compose.yaml file.
  • You should use the names of the Docker compose services (e.g. minio) as the hostname os the services
    • Do not use localhost
    • Alternatively, you could use the IP of the VM
  • There is one aspect that is more difficult to achieve: The creation and configuration of a MinIO bucket to make sure that it is publicly accessible.
    • There are a few ways to achieve this:
      • Configuring bucket policy from Python code
      • Running additional commands inside the Docker container when it is created.

An example of creating a bucket and configuring its access permissions from the Python code: 

  • found = minio_client.bucket_exists(bucket_name)
if not found:
    minio_client.make_bucket(bucket_name)
    policy = {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        "*"
                    ]
                },
                "Action": [
                    "s3:ListBucketMultipartUploads",
                    "s3:GetBucketLocation",
                    "s3:ListBucket"
                ],
                "Resource": [
                    "arn:aws:s3:::images"
                ]
            },
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        "*"
                    ]
                },
                "Action": [
                    "s3:PutObject",
                    "s3:AbortMultipartUpload",
                    "s3:DeleteObject",
                    "s3:GetObject",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": [
                    "arn:aws:s3:::images/*"
                ]
            }
        ]
    }
    minio_client.set_bucket_policy(bucket_name, json.dumps(policy))
    print("Created bucket", bucket_name)

else:
    print("Bucket", bucket_name, "already exists")
    • You may have to change the version number
    • You can also check what the policy looks like with the minio_client.get_bucket_policy("images") command
  • Technically, you do not need to fetch MinIO access and secret key like we did manually in the previous exercises. You should be able to use MINIO_ROOT_USER, and MINIO_ROOT_PASSWORD instead.

Bonus Deliverables:

  • Docker compose file
  • Code, if it required updates
  • Screenshots demonstrating that Docker compose deployment works (from empty state).

Deliverables:

PS! Don't delete the VM and keep services running inside the VM.

  1. Screenshots from exercises
  2. URL (Address including IP and port) to your Flask application
  3. Message board application source code
    • Including the Dockerfile created in exercise 14.4
    • DO NOT include any Python virtual environment folders/files (e.g., .env folder content)
Lahenduste esitamiseks peate olema sisse loginud ja kursusele registreerunud.
  • Arvutiteaduse instituut
  • Loodus- ja täppisteaduste valdkond
  • Tartu Ülikool
Tehniliste probleemide või küsimuste korral kirjuta:

Kursuse sisu ja korralduslike küsimustega pöörduge kursuse korraldajate poole.
Õppematerjalide varalised autoriõigused kuuluvad Tartu Ülikoolile. Õppematerjalide kasutamine on lubatud autoriõiguse seaduses ettenähtud teose vaba kasutamise eesmärkidel ja tingimustel. Õppematerjalide kasutamisel on kasutaja kohustatud viitama õppematerjalide autorile.
Õppematerjalide kasutamine muudel eesmärkidel on lubatud ainult Tartu Ülikooli eelneval kirjalikul nõusolekul.
Courses’i keskkonna kasutustingimused