Practice 4 - Cloud Functions

In this lab we will use IBM Cloud and take a look at the Cloud Function service, which can be considered a Function as a Service (FaaS) or Serverless platform and is based on the Apache OpenWhisk open source project.

In FaaS Coud Computing model your application consists of a set of functions or microservices. Cloud Functions are not constantly running in the background. They are executed only when specific user specified events are triggered (e.g. when new file is uploaded, new entry is added to the database, new message arrives, etc. ).

You will make an account in IBM Cloud and set up multiple Python Cloud Functions which are automatically triggered on either web requests or database events. You will also use the Cloudant NoSQL service as the database for your Cloud Functions. Cloudant is based on the CouchDB open source NoSQL JSON document database.

Additional materials, tutorials and references

• IBM Cloudant documentation - https://cloud.ibm.com/docs/services/Cloudant
• Cloudant Python tutorial - https://cloud.ibm.com/apidocs/cloudant?code=python
• IBM Cloud Functions - https://cloud.ibm.com/docs/openwhisk?topic=cloud-functions-getting-started
• Introduction to JSON - https://www.digitalocean.com/community/tutorials/an-introduction-to-json

Exercise 4.1. Creating an IBM Cloud Account

• Sign up for a free 30 day trial IBM Cloud account at https://cocl.us/ibm_cloud_trial
  • IBM may block UT network from creating accounts. In such a case, try to make an account from a different Wifi or Mobile (through WiFi hotspot) network. (No Need to use VPN for this Lab)
• Activate the account using the link sent by email and log in (verify spam folder or Gmail users check Promotions tab).
• Once logged in, on the left-hand sidebar menu, click on "Resource list"
  • Any cloud resource you have created will appear here in the future. Right now it's most likely empty, but this location will be handy for later tasks.
• Click on "Create a resource"
  • This takes you to the catalog of all available Cloud services on IBM Cloud.
  • Have a quick browse through them to familiarize yourself with the available options.

Exercise 4.2. Creating your first Cloud Function

• From the Catalog, select "Functions" or go to the IBM Cloud Functions console directly at https://cloud.ibm.com/functions//
• Create a new Python Cloud Function
  • On the left-side menu, select Actions, to go to your list of Functions.
  • Click on Create and under Single entities Choose Action
    • Assign a freely chosen Action name to the new action
    • Leave the package as (Default Package)
    • Choose Python 3.9 as the Runtime
  • If you get and error No Cloud Foundry Space choose your Region (London), Cloud Foundry ORG (your username), Cloud Foundry Space (dev)
  • As a result, a simple Hello World function will be generated for you.
• Click Invoke to execute your function to test it.
  • The result should appear after few seconds under "Activations" tab (NB: be patient, can take some seconds)
• You should notice that the result of your function is in JSON format.
  • Input and output of IBM Cloud Functions is JSON by default
  • JSON can be described as a dictionary data structure in JavaScript notation. If you have never used JSON before, read the JSON tutorial here.
  • In Python, JSON objects can also be manipulated as Python dictionaries.

Right now when invoking, we didn't supply any input arguments. Let's add some arguments as a JSON object.

• Modify the function so that it returns a message whose value contains one of the input parameters
  • We can specify the input JSON using the "Invoke with parameters" button.
  • For example, you parameters JSON may look like:

  {
      "name": "John"
  }

  • The parameters are made available to the Python code as the argument of the main function as a dictionary object
    • So, to access the value, you can use dict['key'], where key is the JSON key for your used parameter, such as "name" in above example.
  • An example output would look something like:

Exercise 4.3. Setting up Cloudant Database

We are going to use Cloudant NoSQL database in the following exercises, but first we need to set up a new database instance.

• Go to https://cloud.ibm.com/catalog/services/cloudant and create a new (free) Lite Cloudant NoSQL database.
  • Instance name - choose freely
  • Use default values for the other options
  • You can only have one instance of a Lite plan per service. So if you already have an existing one, you need to delete or modify and proceed.
• The database will be accessible from your IBM cloud Dashboard Resources list
  • You can find it under Services and software
• After creating new Cloudant database wait a few minutes until it finishes initializing, then click on its name to open Cloudant configuration options.
• Create new database credentials
  • Go to Service Credentials -> New credential and assign names for your database credentials.
  • You can view the content of your credentials using the down arrow key attached to your credential.
  • NB! You will need these database credentials(apikey) in the next exercise.

Cloudant is a NoSQL document database. Entries in the database are JSON documents.

• Open the dashboard of your Cloudant database. Go to Manage -> Launch Dashboard
  • Create a new non-partitioned database with the name labdb1 and access it.
  • Create a new JSON document in your DB and add 2 new fields user and message to the document.
    • The document should look something like this ( Do not modify/overwrite the generated "_id" field):

  {
    "_id": "...",
    "user": "Martin",
    "message": "Hello World!"
  }

  • Check the content of the created document.
    • "_id" and "_rev" fields will be automatically generated for each JSON document.

Exercise 4.4. Creating Cloud Function for posting documents to Cloudant database service

In this exercise, we will use a Cloud Function to create a simple web service for submitting data into the Cloudant database.

It is important to note Cloud Functions can be triggered by different kinds of events - web requests, database changes, etc. Input of the Cloud Function will be the content of the event it was triggered on. In case of web requests, input will be the HTTP request and its associated data (e.g. headers, body, html form fields). In case of Cloudant, it is the id of the database document and event type. When you execute the Function directly through the browser, input is usually an empty JSON document.

4.4.1 Create the Cloud Function and set up Cloudant Client

• Return to the Cloud Functions page of IBM Cloud
  • Create a new Cloud Function (Action) called MessageInserter
• In it's python code, add a new function for initializing a client to the database service:

from ibmcloudant.cloudant_v1 import CloudantV1
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator

def get_db_client(params):
    authenticator = IAMAuthenticator(params['apikey'])
    client = CloudantV1(authenticator=authenticator)
    client.set_service_url(params['service_url'])
    return client

This code returns a database client object which we will use to interact with the Cloudant DB. The necessary credentials for connecting to the DB are taken from parameters passed to the function. Let's set them up next.

• Specify database credentials used by get_db_client as additional Cloud Function parameters:
  • Go to the Parameters page under your Function.
  • Add 2 new parameters with correct values based on your own database credentials.
    • service_url - Address of your Cloudant service instance in IBM cloud, find it under Resource list -> Services & Software -> Your Cloudant instance-> External endpoint
    • apikey - Your Cloudant credentials apikey you generated in Ex 4.3.
  • Example of how parameters should look like is provided here:
  • Now these parameters will be automatically added to the input JSON of your Cloud Function and these values can be accessed from inside the function code.
  • This allows us to specify database credentials without hardcoding them into our function code.

4.4.2 Modify the Cloud Function to create new documents and save them to the database

Modify Cloud Function's main method to read the user and message fields and submit a new JSON document containing the values of these fields into the Cloudant database.

• Read the user and message field values from the input document:

def main(params):
    user = params.get("user", "") # 2nd arg is default value if parameter missing
    message = params.get("message", "")

• Create a new JSON document that contains these two fields
  • new_doc = {'message': message, 'user': user}
• Define another new method which uses the DB client to add new documents:

def add_doc_to_db(new_doc, client, db_name="labdb1"):
    return client.post_document(db=db_name, document=new_doc).get_result()

• Use the get_db_client(params) and add_doc_to_db(new_doc, client) methods inside your functions main(params) method to add a new document to the database:

  • client = get_db_client(params)
    modified_doc = add_doc_to_db(new_doc, client) 

  • get_db_client will read apikey and service_url from the additional parameters you specified for the Action.
• Return the document object at the end of main method for ease of debugging:
  • return modified_doc
• Save your Cloud Function

4.4.3 Verify that a new document is created in a database every time the action is invoked with proper inputs.

• As your function is now ready, you need to test it using the Invoke with Parameters
• Open Invoke with Parameters and change action input with a valid JSON document.
  • Example:

  {
  	"message": "Hello",
  	"user": "World"
  }

• After you Invoke the Action you see the output of the document created in Results under Activation and in the database.
• You can debug your code in this section if there is an error in your code.

4.4.4 Modify the Cloud Function to write out proper HTML document
To write out a proper HTML document our cloud function should return a JSON document, which has headers, statusCode and body elements:

• header defines the HTML header values, such as document content type
• Status code can be used to indicate whether the request succeeded and what type of response is being sent
• body contains the content of the HTML document being returned.

To simply return a confirmation that the document was uploaded we can return a JSON document like this at the end of the main() method :

   return {
       "headers": {
         'Content-Type': 'text/html'
       },
       "statusCode": 201,
       "body": '<html><body><h3>Message added to the database</h3></body></html>' 
    }

IBM Cloud function service will convert this into a respective HTTP response which you can see through the Results in Activations Window and the browser(in the next part of this exercise).

4.4.5 Add a web endpoint to your cloud function.
This will make your function publicly accessible from the internet.

• Create a new Web action endpoint
  • Go to Endpoints -> Enable as Web Action
  • This will generate a public web URL for your function that can be accessed from anywhere in the web.
  • NB! You will need this URL in the next step.

4.4.6 Submitting data to the Cloud Function
Let's now create a HTML page for submitting data to your MessageInserter Cloud Function

• Create another new Function (Action), called "MessageBoardHTML". This function will print out an HTML form which can be used by users to enter new messages.
• Let's define the HTML page to be returned by the function as a variable inside the main method:

      form  =  '''
          <html>
            <body>
              <form method="post" action="https://CLOUD_FUNCTION_WEB_ENDPOINT">
                User:<br/>
                <input id="user" name="user" type="text" value=""/> <br/>
                Message:<br/>
                <textarea  rows="10" cols="30" id="message" name="message" type="text" value=""></textarea> <br/>
                <input type="submit" name="submit" value="Send message"   />
              </form>  
            </body>
          </html>
          '''
  

  • Replace https://CLOUD_FUNCTION_WEB_ENDPOINT with the real endpoint URL of the function you previously created in Exercise 4.4.5. (NB! Do not use the endpoint of the function you are editing!)
  • Now let's also modify the function to return a JSON that contains the html document content and also specifies the correct response content-type so that browsers understand that we are dealing with HTML document:

      return {
         "headers": {
           'Content-Type': 'text/html'
         },
         "statusCode": 201,
         "body": form
      }
  

• As a result, the function will return a HTML form which submits user and message fields through a HTML POST request to your web endpoint.
• Create an Endpoint for MessageBoardHTML function as well, similar to the previous exercise.
  • Now, open the endpoint of the MessageBoardHTML function in your browser.
  • Use this HTML form and submit with valid inputs to see if a new document is being created in your database.

4.4.7 Activating log service for the IBM Functions and Cloudant

• To activate log service and to see logs, Click on the left side Log button, to go to https://cloud.ibm.com/observe/logging
• Click on Create Instance button and create a log instance for the London zone
• Go back to https://cloud.ibm.com/observe/logging select the created instance and and click on Configure platform logs
• Choose London region, and the logging instance you just created
• After this, you can go back to the Logging view and open the Dashboard of the logging service to see the log entries page
  • Logs may have delay.
  • Logs will not be saved anywhere because we are using free logging instance. You can only see current, running logs.

Exercise 4.5. Querying and showing existing documents in the database

Let's improve the MessageBoardHTML Function by adding the listing of existing messages.

• Using the same client object from previous exercises, we can fetch documents with:

  • docs = client.post_all_docs(
          db='labdb1',
          include_docs=True,
        ).get_result()
    

  • post_all_docs returns a JSON object, which contains some set of rows from the database (depends on supplied parameters of this function.
• To access the list of rows from the result object:
  • docs['rows'] - gives you an iterable list.
• The actual JSON database document ( which has the fields such as name, message, etc in this example), use the 'doc' on a specific row object, example:

  first_document = docs['rows'][0]['doc']
  first_doc_name = first_document['name']
  

• Based on the above information, update the function so that the 10 most newest documents are queried from the DB, and displayed in the HTML output, alongside the form, similar to the previous lab's Flask application
  • You need to study the post_all_docs documentation to see how to limit # of returned docs, and change the order of results.
  • Don't forget to update the returned HTML to actually display the messages (user, message content)!
  • Tip: first try to get it working with the above "client.post_all_docs(..)" code snippet, before trying to limit to 10 docs and change order of results
  • You should update the MessageInserter function to define a custom id based on some kind of timestamp, instead of letting Cloudant generate the ID automatically like we did so far.
    • i.e., update the construction of new_doc object in that function, pre-specifying a timestamp-based field for key "_id". For instance, you can use epoch time in seconds for id.
    • Because ordering by fields other than the key (ID) is a bit complicated compared to sorting by key (supporetd by post_all_docs). This way, when ordering by key, your messages will also be ordered chronologically.
  • Example output:

Exercise 4.6. Creating a Cloud Function for automatically modifying new documents in the database

Cloud Functions can also be used to automate tasks. We will now create a Cloud Function that is automatically executed for every new document added to the database and which counts how many words and letters the message contained and adds this information as new fields into the the document

• Create a new Python Cloud Function.
  • Go to Actions -> Create -> Create Action
  • Assign NewDocumentTriggerFunction as name to the new function
• Add a Cloudant trigger to your Function.
  • Go to Connected Triggers -> Add Trigger -> Cloudant
  • Assign NewDocumentTrigger as the name of the trigger
  • Choose your database instance under Cloudant Instance
    • If you do not see the Cloudant instance in the list, you will have to configure its location manually by choosing Input your own credentials and then specifying Username, Host, Database and apiKey.
    • You will find correct values for those parameters from Cloudant -> Service Credentials -> View credentials. All values should be without quotation (") marks.
    • Make sure the database is labdb1
  • Click Create & Connect
  • Now your Function will be automatically executed every time there is a new entry added to the labdb1 database and it gets the id of the document as one of the param values as input.
• Modify the NewDocumentTriggerFunction function to compute count of words in the document message field and add a creation-time timestamp to the document
  • Add import statements for Cloudant and time:

    • from datetime import datetime
      from ibmcloudant.cloudant_v1 import CloudantV1
      from ibm_cloud_sdk_core.authenticators import IAMAuthenticator

  • Add the get_db_client(params) method from Exercise 4.4
  • Add a new method to fetch the database object based on its id

  def get_db_doc(doc_id, client, db_name="labdb1"):
  
      return client.get_document(
        db=db_name,
        doc_id=doc_id
      ).get_result()
  

  • Its arguments are document id, and the Cloudant db client
  • Modify the Cloud Function parameters to specify Cloudant service_url and api key (Just like in the Exercise 4.4.1)
  • Modify the def main(params): method to fetch the database object based on its id:

    • client = get_db_client(params)
      doc = get_db_doc(params['id'], client)

    • Id of the modified document will be passed as one of the fields inside the input param object when the Cloudant document modification even is Triggered.
    • However, params object does not contain the rest of the content of the document, which is why we have to use get_db_doc method to fetch it.
  • Calculate the word count based on the document message field (doc['message']).
  • Modify the document by adding new attributes word_count and created_time into it
    • doc['word_count'] = ... #!TODO
    • doc['created_time'] = ... #!TODO
  • Save the modified document to DB, overwriting the original:
    • client.post_document(db='labdb1', document=doc).get_result()
  • Save your Cloud Function
  • Test that the function works by creating new documents in labdb1 database and verifying that word_count attributes are automatically generated for each of them.
    • The result should look something like:

• Modify the function, so that it only computes word_count and created_time once.
  • Our trigger function will be launched every time document is modified. To avoid recursive function triggering, make sure that word_count and created_time are only computed once.
  • Simple way to achieve it is to check whether document already contains created_time field and stop the execution of the function if it does:

    • doc = get_db_doc(params['id'], service)
      if 'created_time' in doc:
         return doc

    • When using IBM Cloud Functions command line interface, it is possible to specify triggers that are only launched on new document creation events.
• Finally, once you've verified the trigger works as expected, update the MessageBoardHTML Function from Ex. 4.5 document listing to also display the word count and creation time.

Bonus Task

• Create a IBM cloud function for converting jpeg images into black and white images.
• Also create a HTML form for uploading image files (use enctype="multipart/form-data") to your cloud function.
• User should be able to use the form to submit a jpeg format image and the result should be returned as a black and white version of the same image.
• PS! PIL image processing library is supported by IBM Functions.
• Form data will be in the input json, converted into base64.
• Getting image data from the form data may be Difficult, as passing an image file to form requires using multipart/form-data
  • Hint: you can use something like this to parse multipart/form-data once you have converted the input into a binary stream (bin_stream):

  • form = cgi.FieldStorage(fp=bin_stream, environ={'REQUEST_METHOD': 'POST', 'CONTENT_LENGTH': bin_length, 'CONTENT_TYPE': param["__ow_headers"]["content-type"]})
    imgdata = form["pic"]
    

Deliverables

• Source code of your Cloud Functions from Exercise 4.4.
  • NB! Do not leave your IBM Cloudant database credentials inside your function code or screenshots!!
  • Provide the URL to the Cloud Function (MessageBoardHTML) web endpoint, which you created in Exercise 4.4.
• Source code of your Cloud Functions from Exercise 4.5.
• Source code of your Cloud Functions from Exercise 4.6.
• Screenshot of your Cloudant labdb1 database view, which should display one of the open documents, show its content(with created_time and word_count).
• Source code (.py and .html) of your Cloud Functions from Bonus Task and Provide the URL to the Cloud Function web endpoint.