Practice 4 - Cloud Functions

In this lab we will use IBM Cloud and take a look at the Cloud Function service, which can be considered a Function as a Service (FaaS) or Serverless platform and is based on the Apache OpenWhisk open source project.

In FaaS Coud Computing model your application consists of a set of functions or microservices. Cloud Functions are not constantly running in the background. They are executed only when specific user specified events are triggered (e.g. when new file is uploaded, new entry is added to the database, new message arrives, etc. ).

You will make an account in IBM Cloud and set up multiple Python Cloud Functions which are automatically triggered on either web requests or database events. You will also use the Cloudant NoSQL service as the database for your Cloud Functions. Cloudant is based on the CouchDB open source NoSQL JSON document database.

Additional materials, tutorials and references

IBM Cloudant documentation - https://cloud.ibm.com/docs/services/Cloudant
Cloudant Python tutorial - https://cloud.ibm.com/apidocs/cloudant?code=python
IBM Cloud Functions - https://cloud.ibm.com/docs/openwhisk?topic=cloud-functions-getting-started
Introduction to JSON - https://www.digitalocean.com/community/tutorials/an-introduction-to-json

Exercise 4.1. Creating an IBM Cloud Account

Sign up for a free IBM Cloud account at https://cloud.ibm.com
- IBM may block UT network from creating accounts. In such a case, try to make an account from a different Wifi or Mobile (through WiFi hotspot) network. (No Need to use VPN for this Lab)
Activate the account using the link sent by email and log in (verify spam folder or Gmail users check Promotions tab).
On the left-hand sidebar menu, click on "Resource list"
- Here, any resource you have created will appear in the future. Right now it's most likely empty.
Click on "Create a resource"
- This takes you to the catalog of all available Cloud services on IBM Cloud. Have a quick browse through them to familiarize yourself with the available options.

Exercise 4.2. Creating your first Cloud Function

Go to the IBM Cloud Functions console at https://cloud.ibm.com/functions//
Create a new Python Cloud Function
- Click on Start Creating and under Single entities Choose Action
- If you get and error No Cloud Foundry Space choose your Region (London), Cloud Foundry ORG (your username), Cloud Foundry Space (dev)
- Assign a freely chosen Action name to the new action
- Leave the package as (Default Package)
- Choose Python 3.7 as the Runtime
- As a result, a simple Hello World function will be generated for you.
Click Invoke to execute your function to test it.
N.B Every time you invoke the action there may be some delay in processing.
- This will execute your Function in the server without any arguments. You can specify default arguments (As a JSON document! ) by using Invoke with parameters -> Change Input link, which can be useful for testing your Cloud Functions.
You should notice that the result of your function is in JSON format.
- Input and output of IBM Cloud Functions is JSON by default
- JSON can be described as a dictionary data structure in JavaScript notation.
- In Python, JSON objects can also be manipulated as Python dictionaries.
- If you have never used JSON before, then read the JSON tutorial here: https://www.digitalocean.com/community/tutorials/an-introduction-to-json
Modify the function output message so it returns a custom message for you instead of the default "Hello World"

Exercise 4.3. Setting up Cloudant Database

We are going to use Cloudant NoSQL database in the following exercises, but first we need to set up a new database instance.

Go to https://cloud.ibm.com/catalog/services/cloudant and create a new (free) Lite Cloudant NoSQL database.
- Instance name - choose freely
- Available authentication methods: - Use both IAM and legacy
- Other Parameters keep as default.
- You can only have one instance of a Lite plan per service. So if you already have an existing one, you need to delete or modify and proceed.
The database will be accessible from your IBM cloud Dashboard: https://cloud.ibm.com/resources
On the Resource List page, go to Services and you will find the Cloudant database you created.
After creating new Cloudant database wait a few minutes till it finishes creating and then click on its name to open Cloudant configuration options.
Create new database credentials
- Go to Service Credentials -> New credential and assign names for your database credentials.
- You can view the content of your credentials using the down arrow key attached to your credential.
- NB! You will need these database credentials(apikey) in the next exercise.
Cloudant is a NoSQL document database. Entries in the database are JSON documents.
Open the dashboard of your Cloudant database. Go to Manage -> Launch Dashboard
- Create a new non-partitioned database with the name labdb1 and access it.
- Create a new JSON document in your database:
  - Add two new fields user and message to the document.
  - The document should look something like this ( Do not don't modify/overwrite the generated "_id" field):
    - ```
    {
      "_id": "...",
      "user": "Martin",
      "message": "Hello World!"
    }
```
- Check the content of the created document.
  - "_id" and "_rev" fields will be automatically generated for each JSON document.

Exercise 4.4. Creating Cloud Function for posting documents to Cloudant database service

In this exercise, we will use a Cloud Function to create a simple web service for submitting data into the Cloudant database.

It is important to note that input to the Cloud Function will be the content of the event it was triggered on. In case of web requests, input will be the HTTP request and its associated data (e.g. headers, body, html form fields). In case of Cloudant, it is the id of the database document and event type. When you execute the Function directly through the browser, input is usually an empty JSON document.

4.4.1 Create another Cloud Function
We will modify the function to create a new document in labdb1 database every time the Function is executed.

Open the Page where you created an action.
Add a new method for putting documents into your database:
- First import Cloudant API
  - from cloudant.client import Cloudant
- Then add a new add_doc_to_db(new_doc, username, apikey) method to your Cloud Function code
```
def add_doc_to_db(new_doc, username, apikey):
    db_name = "labdb1" 
    client = Cloudant.iam(username, apikey, connect=True)
    my_database = client[db_name]
    return my_database.create_document(new_doc)
```
- Specify database credentials as additional Cloud Function parameters:
  - Go to the Parameters page under your Function.
  - Add 2 new parameters with correct values based on your own database credentials.
    - username - Your username from Launch Dashboard of Cloudant -> database name -> Permissions -> Copy the Cloudant username on labdb1
    - apikey - Your Cloudant credentials apikey you generated in Ex 4.3.
  - Example of how parameters should look like is provided here: parameters.png
  - Now these parameters will be added to the input (document) of your Cloud Function and these values can be accessed from inside the function code.
  - This allows us to specify database credentials without hardcoding them into our function code.

4.4.2 Modify the Cloud Function to create new documents and save them to the database
Modify Your Cloud Function to read the user and message fields and submit a new JSON document containing the values of these fields into the Cloudent database.

Read the user and message field values from the input document:

def main(param):
    user = ""
    if 'user' in param:
        user = param['user']

    message = ""
    if 'message' in param:
        message = param['message']

Create a new JSON document that contains these two fields
- new_doc = {'message': message, 'user': user}
Use the add_doc_to_db(new_doc, username, apikey) method inside your functions main(param) method to add a new document to the database:
- modified_doc = add_doc_to_db(new_doc, param['username'], param['apikey'])
- username and apikey will be read from the additional parameters you specified for the Action.
Return the document object at the end of main method for ease of debugging:
- return modified_doc
Save your Cloud Function

4.4.3 Verify that a new document is created in a database every time the action is invoked with proper inputs.

As your function is now ready, you need to test it using the Invoke with Parameters
Open Invoke with Parameters and change action input with a valid JSON document.
Example:

{
	"message": "Hello",
	"user": "World"
}

After you Invoke the Action you see the output of the document created in Results under Activation and in the database.
You can debug your code in this section if there is an error in your code.

4.4.4 Modify the Cloud Function to write out proper HTML document
To write out a proper HTML document our cloud function should return a JSON document, which has headers, statusCode and body elements:

header defines the HTML header values, such as document content type
Status code can be used to indicate whether the request succeeded and what type of response is being sent
body contains the content of the HTML document being returned.

To simply return a confirmation that the document was uploaded we can return a JSON document like this at the end of the main() method :

   return {
       "headers": {
         'Content-Type': 'text/html'
       },
       "statusCode": 201,
       "body": '<html><body><h3>Message added to the database</h3></body></html>' 
    }

IBM Cloud function service will convert this into a respective HTTP response which you can see through the Results in Activations Window and the browser(in the next part of this exercise).

4.4.5 Add a web endpoint to your cloud function.
This will make your function publicly accessible from the internet.

Create a new Web action endpoint
- Go to Endpoints -> Enable as Web Action
- This will generate a public web URL for your function that can be accessed from anywhere in the web.
- NB! You will need this URL in the next step.

4.4.6 Submitting data to the Cloud Function
Let's now create a html page for submitting data to your Cloud Function

Modify the Function you created in the Exercise 4.2 to also print out an HTML form which can be used by users to enter new messages.

Let's define the html page to be returned by the function as a variable inside the main method:

    form  =  '''
        <html>
          <body>
            <form id="form_1" method="post" action="https://CLOUD_FUNCTION_ENDPOINT">
              User: <input id="user" name="user"  size="30" type="text" value=""/> <br />
              Message: <textarea  rows="10" cols="30" id="message" name="message" type="text" value=""></textarea> <br />
              <input id="saveForm" type="submit" name="submit" value="Send message" />
            </form>	
          </body>
        </html>
        '''

Replace https://CLOUD_FUNCTION_ENDPOINT with the real endpoint URL of the function you previously created in Exercise 4.4.5. (NB! Do not use the endpoint of the function you are editing!)
Now let's also modify the function to return a JSON that contains the html document content and also specifies the correct response content-type so that browsers understand that we are dealing with HTML document:

    return {
       "headers": {
         'Content-Type': 'text/html'
       },
       "statusCode": 201,
       "body": form
    }

As a result, the function will return a HTML form which submits user and message fields through a HTML POST request to your web endpoint.
- Open the endpoint of the modified function in your browser. (NB! Use the endpoint of the function you just modified! (Exercise 4.2 function) )
Use this HTML form and submit with valid inputs to see if a new document is being created in your database.

4.4.7 Activating log service for the IBM Functions and Cloudant

To activate log service and to see logs, Click on the left side Log button, to go to https://cloud.ibm.com/observe/logging
Click on Create Instance button and create a log instance for the London zone
Go back to https://cloud.ibm.com/observe/logging and click on Configure platform logs
Choose London region, and the logging instance you just created
After this, you can go back to the Logging view and click on View LogDNA to see the log entries page
- Logs may have delay.
- Logs will not be saved anywhere because we are using free logging instance. You can only see current, running logs.

Exercise 4.5. Creating a Cloud Function for automatically modifying new documents in the database

Cloud Functions can also be used to automate tasks. We will now create a Cloud Function that is automatically executed for every new document added to the database and which counts how many words and letters the message contained and adds this information as new fields into the the document

Create a new Python Cloud Function.
- Go to Actions -> Create -> Create Action
- Assign lab4dbTriggerFunction as name to the new function
Add a Cloudant trigger to your Function.
- Go to Connected Triggers -> Add Trigger -> Cloudant
- Assign lab4dbTrigger as the name of the trigger
- Choose your database instance under Cloudant Instance
  - If you do not see the Cloudant instance in the list, you will have to configure its location manually by choosing Input your own credentials and then specifying Username, Host, Database and apiKey.
  - You will find correct values for those parameters from Cloudant -> Service Credentials -> View credentials. All values should be without quotation (") marks.
  - Make sure the database is labdb1
- Click Create & Connect
- Now your Function will be automatically executed every time there is a new entry added to the labdb1 database and it gets the id of the document as one of the param values as input.
Modify the lab4dbTriggerFunction function to compute both count of letters and words in the document message field
- Add import statements for Cloudant and time:
  - from datetime import datetime
- Add a new method to fetch the database object based on its id
```
def get_db_doc(doc_id, client):
    document = client.get_document(
      db='labdb1',
      doc_id=doc_id
    ).get_result()

    return document
```
- Its arguments are document id and the DB client object similar to previous exercises
- Modify the Cloud Function to include the DB client initialization (Just like in the Exercise 4.4.1)
- Modify the def main(params): method to fetch the database object based on its id:
  - doc = get_db_doc(params['id'], client)
  - Id of the modified document will be passed as one of the fields inside the input params object when the Cloudant document modification even is Triggered.
  - However, params object does not contain the rest of the content of the document, which is why we have to use get_db_doc method to fetch it.
- Calculate letter and word count based on the document message field (doc['message']).
- Modify the document by adding new attributes word_count and letter_count into it
  - doc['word_count'] = ...
- Save the modified document:
  - doc.save()
- Save your Cloud Function
- Test that the function works by creating new documents in labdb1 database and verifying that word_count and letter_count attributes are automatically generated for each of them.
  - The result should look something like:

Modify the function, so that it only computes word_count and letter_count once.
- Our trigger function will be launched every time document is modified. To avoid recursive function triggering, make sure that word_count and letter_count are only computed once.
- Simple way to achieve it is to check whether document already contains word_count field and stop the execution of the function if it does:
  - ```
  doc = get_db_doc(param)
  if 'word_count' in doc:
     return doc
```
- When using IBM Cloud Functions command line interface, it is possible to specify triggers that are only launched on new document creation events.

Bonus Task

Create a IBM cloud function for converting jpeg images into black and white images.
Also create a HTML form for uploading image files (use enctype="multipart/form-data") to your cloud function.
User should be able to use the form to submit a jpeg format image and the result should be returned as a black and white version of the same image.
PS! PIL image processing library is supported by IBM Functions.
Form data will be in the input json, converted into base64.
Getting image data from the form data may be Difficult, as passing an image file to form requires using multipart/form-data
- Hint: you can use something like this to parse multipart/form-data once you have converted the input into a binary stream (bin_stream):
- ```
form = cgi.FieldStorage(fp=bin_stream, environ={'REQUEST_METHOD': 'POST', 'CONTENT_LENGTH': bin_length, 'CONTENT_TYPE': param["__ow_headers"]["content-type"]})
imgdata = form["pic"]
```

Deliverables

Source code of your Cloud Functions from Exercise 4.4.
- NB! Do not leave your IBM Cloudant database credentials inside your function code or screenshots!!
- Provide the URL to the Cloud Function web endpoint, which you created in Exercise 4.4.
Source code of your Cloud Functions from Exercise 4.5.
Screenshot of your Cloudant labdb1 database view, which should display one of the open documents, show its content(with letter_count and word_count).
Source code (.py and .html) of your Cloud Functions from Bonus Task and Provide the URL to the Cloud Function web endpoint.

4. lab 4

Solutions for this task can no longer be submitted.

Cloud Computing 2020/21 spring