Cloud machine learning (ML) APIs

How to use cloud-based machine learning services with RPA Framework.

Cloud APIs

Supporting advanced data analytics or decision logic beyond simple deterministic rules often requires machine learning (ML) solutions. One way to provide the needed intelligence is to use commercial cloud services that can be applied for a number of typical use cases. Such services include, for example, image analysis, natural language processing, document analysis, and optical character recognition (OCR).

This tutorial demonstrates how to use cloud-based ML APIs with RPA Framework. Currently, a selected set of services from AWS (RPA.Cloud.AWS), Microsoft Azure (RPA.Cloud.Azure) and Google Cloud (RPA.Cloud.Google) are available.

You can access cloud ML services that are not supported by RPA Framework by writing custom Robot Framework libraries to connect them via REST APIs or service-specific SDKs. If your RPA team has in-house machine learning expertise, the possibilities for self-developed ML services from the cloud are almost limitless. A separate article on this topic will be published later.

Prerequisites

To use cloud ML APIs, you need:

  • An account for the cloud provider of your choice.
  • The required ML service(s) activated and accessible from the public internet.
  • Credentials to the service(s) - the format depends on the cloud provider.

Set up your development environment. Specifically, install the RPA Framework package with the optional aws or google dependencies, depending on which cloud providers you are planning to use:

pip install --upgrade "rpaframework[aws,google]"

Initialize the software robot directory

Navigate to your projects directory in the terminal or the command prompt. Initialize the software robot directory:

robo init ml-cloud-apis

Navigate to the directory:

cd ml-cloud-apis

Note that Robocode Lab users can create a new activity into the projects directory with the GUI.

Modify config/conda.yaml to include AWS and Google dependencies:

dependencies:
  - pip:
      - rpaframework[aws,google]==1.*

Setup the access keys

The authentication method varies depending on the cloud service:

  • AWS: AWS key and AWS key ID (AWS_KEY and AWS_KEY_ID)
  • Azure: Azure subscription key (AZURE_SUBSCRIPTION_KEY)
  • Google Cloud: Service credentials file

There are three different ways to set the access credentials - See the documentation of the RPA.Cloud libraries. Here we present an approach that works both in a local and Robocloud setting.

For local robot runs, we use a vault.json file placed outside the repository, e.g., in /tmp/vault.json. For AWS and Azure, we can save the key values directly in the file. For Google Cloud, the content of the service credentials file must be mapped to a string, with special characters escaped. This can be done, e.g., with the following Python script.

import json

vault = {}

vault["aws"] = {"AWS_KEY_ID": "aws-access-key-id",
                "AWS_KEY": "aws-secret-access-key"}

vault["azure"] = {"AZURE_SUBSCRIPTION_KEY": "azure-subscription-key"}

with open("path-to-gcp-credentials-file", 'r') as f:
    gcp_json_str = f.read()

vault["gcp"] = {"json_content": gcp_json_str}

with open('/tmp/vault.json', 'w') as f:
    json.dump(vault, f)

This will result in the following JSON file:

{
  "aws": {
    "AWS_KEY_ID": "aws-access-key-id",
    "AWS_KEY": "aws-secret-access-key"
  },
  "azure": { "AZURE_SUBSCRIPTION_KEY": "azure-subscription-key" },
  "gcp": { "json_content": "GCP service credentials file as string" }
}

If the activity is run in Robcloud, store the above credential information in your workspace vault with the names aws, azure and gcp.

Configure file vault support

To run in your local or development environment, paste the following in devdata/env.json:

{
  "RPA_SECRET_MANAGER": "RPA.Robocloud.Secrets.FileSecrets",
  "RPA_SECRET_FILE": "/tmp/vault.json"
}

Sample data

In this tutorial, we will use images of an invoice and a sphygmomanometer to showcase the image processing functionalities. For natural language processing, we will be working on a piece of text given as a string. These can be defined in the variables/variables.py file.

import os
import json

# Folder paths
VAR_DIR = os.path.dirname(os.path.abspath(__file__))
ROOT_DIR = os.path.join(VAR_DIR, "..")
DATA_DIR = os.path.join(ROOT_DIR, "output")

# Sample data
INVOICE_URL = "https://templates.invoicehome.com/invoice-template-us-neat-750px.png"
INVOICE_FILE = os.path.join(DATA_DIR, "invoice.png")
PICTURE_URL = "https://upload.wikimedia.org/wikipedia/commons/9/9d/Beurer_BM26_sphygmomanometer.jpg"
PICTURE_FILE = os.path.join(DATA_DIR, "picture.jpg")
TEXT_SAMPLE =  (
    "A software robot developer creates digital agents for robotic process "
    "automation (RPA), test automation, application monitoring, or some "
    "other use. Tens of thousands of new jobs are predicted to be created "
    "in the RPA industry. Most of these will be for developers. "
    "The demand for software robot developers is growing. Many companies "
    "will employ teams of software robot developers to build and operate "
    "their automated workforce. Other organizations hire external "
    "developers to offer them automation with a 'robotics-as-a-service' model.")

The sample invoice and picture are shown below.

invoice picture

Using ML services with RPA Framework

The robot code resides in the tasks/robot.robot file. In the *** Settings *** section, we define which libraries are used and configure the cloud libraries.

*** Settings ***
Documentation   ML API examples
Library         RPA.FileSystem
Library         RPA.HTTP
Library         RPA.Tables
Library         RPA.Cloud.AWS
...             region=us-east-1
...             robocloud_vault_name=aws
Library         RPA.Cloud.Azure
...             robocloud_vault_name=azure
Library         RPA.Cloud.Google
...             robocloud_vault_name=gcp
...             robocloud_vault_secret_key=json_content
Variables       variables.py

The first task is to download the sample images:

*** Tasks ***
Download sample files
   Download   ${INVOICE_URL}   target_file=${INVOICE_FILE}   overwrite=True
   Download   ${PICTURE_URL}   target_file=${PICTURE_FILE}   overwrite=True

The basic functionality of RPA.Cloud libraries is to send a request to the API and return the response as JSON without any modification to the content. The JSON data can then be processed further based on the needs of the automated process. Such operations are not currently fully supported by RPA Framework since each service has a different response syntax.

AWS

To analyze a scanned invoice, we use the Textract service from AWS.

*** Tasks ***
Analyze invoice with AWS Textract and find tables from the response
    Init Textract Client  use_robocloud_vault=True
    ${response}=          Analyze Document  ${INVOICE_FILE}  ${DATA_DIR}${/}textract.json
    ${tables}=            Get Tables
    FOR   ${key}   IN   @{tables.keys()}
       Write Table To Csv   ${tables["${key}"]}   ${DATA_DIR}${/}table_${key}.csv
    END

After configuring the client and getting the answer from AWS, we use the Get Tables keyword to collect tables from the response. In this example, table data is correctly found.

QTYDESCRIPTIONUNIT PRICEAMOUNT
1Front and rear brake cables100.00100.00
2New set of pedal arms15.0030.00
3Labor 3hrs5.0015.00
Subtotal145.00
Sales Tax 6.25%9.06
TOTAL$154.06

The full response from AWS is stored in a JSON file (here output/textract.json). Parsing the other data, like key-value pairs, from the JSON response needs to be done by separate functions. The syntax of the Textract response and code examples are available on the Textract developer guide.

Azure

The usage of (RPA.Cloud.Azure) is demonstrated by Text analytics service. In the following task, the robot recognizes the language, key phrases, and sentiment of a sample text.

*** Tasks ***
Analyze text sample with Azure
    Init Text Analytics Service   use_robocloud_vault=True
    Detect Language     Vilken språk talar man in Åbo?  ${DATA_DIR}${/}text_lang.json
    Key Phrases         ${TEXT_SAMPLE}  ${DATA_DIR}${/}text_phrases.json
    Sentiment analyze   ${TEXT_SAMPLE}  ${DATA_DIR}${/}text_sentiment.json

The outcome of the analysis is stored in JSON files (here output/text_lang.json, output/text_phrases.json, and output/text_sentiment.json). As a summary, we find the following information:

detectedLanguageconfidenceScore
Swedish1.0
keyPhrases
'teams of software robot developers' 'external developers', 'test automation', 'robotic process automation', 'RPA industry', 'digital agents', 'application monitoring', 'Tens of thousands', 'new jobs', 'companies', 'demand', 'service', 'use', 'robotics', 'organizations', 'automated workforce', 'model'
sentimentconfidenceScores
neutral{'positive': 0.03, 'neutral': 0.96, 'negative': 0.01}

Google Cloud

From RPA.Cloud.Google we try Google Vision AI which can be used, for example, to label an image or detect the text.

*** Tasks ***
Analyze image with Google Vision AI
   Init Vision Client   use_robocloud_vault=True
   ${labels}=    Detect Labels   ${PICTURE_FILE}   ${DATA_DIR}/vision_labels.json
   ${text}=      Detect Text   ${PICTURE_FILE}   ${DATA_DIR}/vision_text.json
   Log   ${DATA_DIR}

The full responses are stored in JSON files (here output/vision_labels.jsonand output/vision_text.json). The summary of the outcome is shown below. Note that in order to get "sphygmomanometer" among the labels, one needs to train a model since such a term is not included in Google Vision AI. The recognized text blocks cover almost all the text appearing in the image. The full response data also contains the bounding box information.

LabelScore
Electronic device0.8969995
Technology0.89112115
Measuring instrument0.827221
Arm0.75604373
Cyclocomputer0.53043586
Pedometer0.5304172
Text
"beurer", "102", "63", "LINKER ARM", "LEFT ARM", "WHO", "SYS", "mmHg", "mmHg", "DIA", "mmHg", "2-3 cm", "Arterie", "artery", "LATEX", "12 20M" "9.05.", "FREE", "PULSE", "/min", "22-35 cm", "Item No.: 162.972", "urer GmbH, Söflinger Strasse 218, 89077 Ulm, Germany", "M+C"; "Set"

Wrap and run the robot

If using Robocode CLI, wrap and run the robot:

robo wrap --force
robo run entrypoint.sh -v devdata/env.json