Practical Session 1: Working with Cloud Infrastructure (ETAIS)

The aim of the practical session is to learn the basics of cloud computing that serve as fundamental infrastructure to deploy your application services in a scalable manner. Here, you will learn how to provision virtual machines from the cloud infrastructure and access them publically by managing the security groups. In this experiment, we will use a cloud infrastructure service known as Estonian Scientific Computing Infrastructure (ETAIS) managed by the Estonian Scientific Community (https://etais.ee/about/). This cloud infrastructure is configured on large-scale hardware resources using the OpenStack cloud operating system.

ETAIS project is being carried out by a consortium of four institutions: the University of Tartu, Tallinn University of Technology, National Institute of Chemical Physics and Biophysics, and The Education and Youth Board (Harno).

How to access ETAIS cloud infrastructure from the University of Tartu network:

To access the local university cloud resources your computer has to be inside the Institute network. So you should either use lab computers, Eduroam Wifi (inside the institute building) or set up a VPN connection to the university network.

VPN (English on the right side column) - https://wiki.ut.ee/pages/viewpage.action?pageId=17105590
Eduroam Wifi (English on the right side column) - https://wiki.ut.ee/display/AA/Eduroam

Practical session communication!

There will be physical lab sessions, but it is encouraged to attend online for the foreseeable future.

Lab supervisors will provide support through Slack
We have set up a Slack workspace for lab and course-related discussions. The invitation link was sent separately to all students through email.
- Use the #practice1-etais Slack channel for lab-related questions and discussion.
When asking questions or support from lab assistants, please make sure to also provide all needed information, including screenshots, configurations, and even code if needed.
- If code needs to be shown to the lab supervisor, send it (or a link to it) through Slack Direct Messages.
- If you have not received a Slack invitation, please contact lab assistants through email.

In case of issues check:

Pinned messages in the #practice1-etais Slack channel.
Possible solutions to common issues section at the end of the guide.
Ask in the #practice1-etais Slack channel.

Part 1. Introduction to ETAIS cloud infrastructure.

Estonian Scientific Computing Infrastructure (ETAIS) belongs to the Estonian roadmap of research infrastructures providing computing and storage resources for the Estonian Scientific Community. ETAIS aims to increase the competitiveness of the Estonian computing and data-intensive research disciplines by providing access to a new and modern scientific computing infrastructure. The OpenStack cloud operating system is deployed on a large set of hardware infrastructure located at universities including the University of Tartu.

For more information you can refer to www.etais.ee We have added a presentation video on a brief introduction to ETAIS and how to log in and provision the resources using a web interface.

Introductory video presentation on ETAIS

Part 2. Working with ETAIS cloud infrastructure

In these exercises, you will learn to access the ETAIS web interface and provision virtual machines using the ETAIS web interface.

Exercise 2.1. Accessing the ETAIS cloud services

The following steps will guide you to connect to the ETAIS web interface.

Log into https://minu.etais.ee and click on Sign in with TAAT.
From the drop-down, select University of Tartu and log in using your university username and password.
- Contact lab supervisor if you have issues while logging in.
Familiarize yourself with the available ETAIS web functionality as mentioned in Part 1.
Look for a button called "Select workspace" (up and center) and select workspace DevOps2021Fall (If you have a problem in accessing the workspace then please report to teaching assistants in the slack channel!)
Create an ssh Key Pair for accessing Virtual Machines over the network. Make sure the name of the Key Pair includes your last name!
- Firstly, create keys(SSH with RSA 4096 bits) in your machine/laptop using the following tools:
  - Linux: ssh-keygen
  - Windows: Puttygen (Store the private key with extension .ppk)
  - Mac OS: ssh-keygen
- Now, in the ETAIS web interface, on the right side click on Details --> SSH keys --> Add Key and complete the steps. (Make sure to keep key last name same as a private key)

Exercise 2.2. - Requesting computing resources from the cloud

In this exercise, you will start a Cloud instance (or virtual machine) while specifying its configuration and computing resources available for it.

Click on the "Resources" tab and then "VMs" in the left side panel and go to "Add Resource" and select "Virtual Machine in DevOps-cloud". Add the following values to the offering configuration page.
- VM Name --> Give your last name and a random number, e.g. dehury-4546 or poojara-34
- Click on "Show choices" in images and select Ubuntu 18.04 virtual machine image.
- Click on "Show choices" in Flavour and select m3.tiny.
- System volume size --> 10 GB
- System Volume type --> prod (HPC production HDD), leave other details as default.
- SSH Public key--> select your recently created key.
- Security Groups --> default, ping, web, ssh.
- Networks --> leave it by default, Next select Auto-assign floating IP (to assign an IP to access machine in UT network)
Finally, click on "Add to cart".
It will take to the purchase page and now click on "purchase".
After few seconds, click on "Refresh" and you should see the state-->done condition.

Exercise 2.3. Accessing your Cloud instance using ssh

We will use the Secure Shell (ssh) protocol to log into the started instance over the network. Instances in the cloud can have multiple IP addresses. External IP for accessing the instance from outside the cloud and Internal IP for accessing the instance from inside the cloud (from other instances). However, our instances have two IP's in the current configuration and note down "External IP" for connecting to the VM.
You have to be in the university network to access the VMs. Use VPN or log in to eduroam, if you're using your own laptop

Log into the instance through ssh using SSH Key-based authentication
- On Linux:
  - ssh -i path_to_my_key_pair_file ubuntu@<instance public ip address>
  - For example: ssh -i .cloud/shiva_key.pem ubuntu@172.17.64.63
  - if you get an error, check that the path to the keyfile is correct and that it has correct rights (chmod 400 <filename>)
- On Windows:
  - Use Git BASH command line, Putty, WinSCP, or SSH secure shell program to get a command-line interface to a remote server through ssh.
  - When using Git BASH, the command is exactly the same as in the Linux command line:
    - ssh -i path_to_my_key_pair_file ubuntu@<instance public IP address>
  - When using Putty:
    - On the Putty startup screen enter your External IP.
    - Click on the Connection menu, then choose Data.
      - Enter the default login of the user - ubuntu.
    - Choose Connection -> SSH -> AUTH and browse for private key you generated earlier.
    - Click Open.

If you are unable to access the machine over ssh then modify and enable port number (ssh) in security group section

Exercise 2.4. Deploying an application

In this exercise, you're going to learn how to deploy an application in this virtual machine. The sample web application is to read IoT sensor data from the CSV file and display it in the form of a table on the HTML page. We will use the python flask framework to create this web application.

update apt using sudo apt-get update
Install python virtual enviornment sudo apt install python3-venv.
Create a directory for app mkdir flask_app && cd flask_app
Create virtual enviornment python3 -m venv venv
Activate virtual enviornment source venv/bin/activate
Install flask fraemwork pip install Flask
Install pandas framework pip install pandas
Download the flask project with required code using git clone https://github.com/shivupoojar/webaspp-flask.git
Following is a main python code:

    #Flask imports
from flask import Flask, render_template, send_file, make_response, url_for
from flask import Response

#Pandas 
import pandas as pd


import io
import random

# Read the data 
df  = pd.read_csv("CO2.csv",error_bad_lines=False)

# Convert time format from unix milliseconds to datetime64 format
df['time'] = df['time'].astype('datetime64[ms]')


# Create a flask app
app = Flask(__name__)

# main route
@app.route('/')
@app.route('/pandas', methods=("POST", "GET"))
def GK():
    return render_template('home.html',
                           PageTitle = "plot",table=[df.to_html(classes='data', index = False)], titles=df.columns.values)

if __name__ == '__main__':
    app.run(debug = True,host='0.0.0.0',port=5000)

Add your name to the HTML page
Now run the application python3 app.py
Now open the application using http://VM_IP:5000 in your machine browser

Take a screenshot of your webpage. Note!! your name and IP address should be visible

NB! Once you are done, you must delete your instance and the Volume you created! Also, be careful you do not delete the work of other students.

Deliverables:

Your instance must have been terminated (deleted)!
Screenshot created in exercise 2.4

Deadline: 17th of September

1. Lab 1

Solutions for this task can no longer be submitted.

Possible solutions to common issues

If you can not access the Cloud instance over SSH,
- it may be because you have not set up a VPN connection.
If you can not access your instance over port 5000
- it may be because you have not opened port 5000 in your security group (add allow-all security rule)
- or added the created the security group to your instance
- or not used correct IP filter for Remote IP Prefix:
  - Remote IP Prefix should stay 0.0.0.0/0 (this means all devices from any IP address can access this port)
  - Do not assign Remote Security Group, it would limit to the specified port from only the cloud instances inside the selected security group.
If you are trying with Centos7 image, you will face issues with Exercise 2.4

DevOps: Automating Software Delivery and Operations 2021/22 fall