Overview

The Cloud Retail service and the Retail API enables customers to build end-to-end personalized recommendation systems without requiring a high level of expertise in machine learning, recommendation system, or Google Cloud.

In this lab, you will explore the monitoring and management features available for the Retail service and examine some common issues and errors associated with importing and updating catalog and event data. The performance of the Recommendations AI and Product Search services is dependent on having an error free stream of user events so it is important to monitor these services and create alerts that trigger when there are issues with your Retail catalog and user event data.

This lab uses a subset of the Google Merchant Center dataset for product catalog and user event data.

Objectives

In this lab, you will learn how to complete the following tasks:

• Review the Retail dashboard and explore the monitoring metrics.
• Create and configure alerts for Retail services.
• Review data ingestion and Cloud Logging logs for Retail.
• Update product data in the catalog using the Retail API.

Setup and requirements

Qwiklabs setup

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

1. Sign in to Qwiklabs using an incognito window.

2. Note the lab's access time (for example, 1:15:00), and make sure you can finish within that time.
   There is no pause feature. You can restart if needed, but you have to start at the beginning.

3. When ready, click Start lab.

4. Note your lab credentials (Username and Password). You will use them to sign in to the Google Cloud Console.

5. Click Open Google Console.

6. Click Use another account and copy/paste credentials for this lab into the prompts.
   If you use other credentials, you'll receive errors or incur charges.

7. Accept the terms and skip the recovery resource page.

Start Cloud Shell

While in Google Cloud you can operate Google Cloud remotely from your own machine. This lab uses both the Google Cloud Console and the Cloud Shell, a command line environment running in Google Cloud.

1. From the Cloud Console, click Activate Cloud Shell.

   

   Note: If you've never started Cloud Shell before, you are presented with an intermediate screen describing what it is. If that's the case, click Continue and you won't ever see it again.

   Here's what that one-time screen looks like:

   

   It should only take a few moments to provision and connect to Cloud Shell.

   Cloud Shell provides you with terminal access to a virtual machine hosted in the cloud. The virtual machine includes all the development tools that you'll need. It offers a persistent 5GB home directory and runs in Google Cloud, greatly enhancing network performance and authentication. Much, if not all, of your work in this lab can be done through the Cloud Console and Cloud Shell using only a browser.

   Once connected to Cloud Shell, you should see that you are already authenticated and that the project is already set to your project ID.

2. Run the following command in Cloud Shell to confirm that you are authenticated:

   gcloud auth list

   Output:

   Credentialed Accounts ACTIVE: * ACCOUNT: {{{user_0.username| Lab User Name}}}

3. To set the active account, run:

   gcloud config set account {{{user_0.username| Lab User Name}}} Note: The gcloud command-line tool is the powerful and unified command-line tool in Google Cloud. It comes preinstalled in Cloud Shell. Among its features, gcloud offers tab completion in the shell. For more information, refer to the gcloud CLI overview guide.

4. Run the following command to confirm that you are using the correct project for this lab:

   gcloud config list project

   Output:

   [core] project = {{{project_0.project_id | Project ID}}}

5. If the correct project is not listed, you can set it with this command:

   gcloud config set project {{{project_0.project_id| Project ID}}}

   Output:

   Updated property [core/project].

Task 1. Enable the Retail API

Before you can begin using the Retail Recommendations AI or Retail Search APIs, you must enable the Retail API.

1. On the Navigation menu (), click VIEW ALL PRODUCTS. In the Artificial Intelligence section, click Search for Retail.

2. Click Turn On API.

3. Click Continue and accept the data terms by clicking Accept.

4. Click Get Started.

Task 2. Import product catalog and user event data

Import Retail products schema data from BigQuery

In this task, import product data into the catalog from a BigQuery table that uses the Retail products schema.

1. In the GCP Console, on the Navigation menu (), click VIEW ALL PRODUCTS. In the Artificial Intelligence section, click Search for Retail > Data to open the Retail Data management page.

2. Make sure the Catalog tab is selected and click Import.

3. Configure the import parameters as follows to import the product catalog:

   • For Import type, select Product Catalog
   • For Source of data, select BigQuery
   • For Import Branch, select Branch 0

4. For Big Query table, click Browse.

5. Enter products in the search box and click Search.

6. Select the radio button for products Dataset: retail table.

7. Click Select.

Note: If you click the table name you will open the Data Catalog page and will need to return to the Retail products import page.

8. Click Import.

Note: If you get an exception 'A service account email is required to schedule auto-import', You can use the username as service account email.

You need to wait for a pop-up message to appear with a message similar to the following:

Successfully scheduled import operation import-products-6583047802807380211. It may take up to 5 minutes to see your new long running operation in the Integration Activity panel.

When the import task is scheduled you will also see the details of a gcloud scheduler command displayed that you can use to schedule a regular data import task.

9. Click X to close the popup that appeared to tell you that the import operation was successfully scheduled.

10. Click Cancel to close the import page and return to the Retail Data page to check the status of your catalog data import task.

11. In the Search Retail navigation menu, click Data and then click Activity Status to monitor the progress of the import task.

The import task will take a minute or two for the status of the import task in the Product catalog import activity status section to change to Succeeded. A total of 1268 items will have been imported.

Import user event data from Cloud Storage

In this task, import user event data from a BigQuery table.

1. In the GCP Console, on the Navigation menu (), click VIEW ALL PRODUCTS. In the Artificial Intelligence section, click Search for Retail > Data to open the Retail Data management page.

2. Make sure the Event tab is selected and click Import.

3. Configure the import parameters as follows to import the product catalog:

   • For Import type, select User Events
   • For Source of data, select Google Cloud Storage

4. For Google Cloud Storage location, click the Browse button.

5. Navigate to the storage bucket called and select the file recent_retail_events.json.

6. Click the Filename to make sure it is selected.

7. Click Select.

8. Click Import.

You need to wait for a pop-up message to appear with a message similar to the following:

Successfully scheduled import operation import-products-6583047802807380211. It may take up to 5 minutes to see your new long running operation in the Integration Activity panel

When the import task is scheduled you will also see the details of a gcloud scheduler command displayed that you can use to schedule a regular event import task.

9. Wait for the import task to be scheduled with the gcloud scheduler command displayed.

10. Click X to close the popup that appeared to tell you that the import operation was successfully scheduled.

11. Click Cancel to close the import page and return to the Retail Data page to check the status of your event data import task.

12. In the Retail navigation menu, click Data and then click Activity Status to monitor the progress of the import task.

The import task will take a minute or two for the status of the import task in the User events import activity status section to change to Succeeded. Approximately 32,000 items will have been imported and 5 items will have failed.

Task 3. Review the Retail dashboard and explore the monitoring metrics

In this task, you will explore the features in the Retail Dashboard and the information provided.

1. In the GCP Console, on the Navigation menu (), click VIEW ALL PRODUCTS. In the Artificial Intelligence section, click Search for Retail > Dashboard to open the Retail dashboard.

The Dashboard contains high-level status information and quick links for the following:

• Getting Started: Links for importing product data, importing user event data, and creating serving configs.
• Credentials: List of active API keys with access to this project's Retail API.
  • There are no active keys because we use a service account and access tokens for access in this lab.
• Catalog: Summary information about the total number of products in and out of stock in your catalogs.
  • There are 746 products in stock and 1,268 products in total in your default catalog branch.
• Event: Summary information about event data over the past 90 days.
  • There are ~32,000 events, with just under 2% of events showing as unjoined where the event could not be linked to a product in the catalog.
• Errors: Summary of Retail API errors for catalog updates, events, and Recommendations AI models.
  • Most of the summary errors will show green but there are 9 import errors in the last 24 hours, 4 for unjoined events and 5 that resulted from incorrect schema that you explored in an earlier lab.
• Recommended Alerts: Link to a pop-out to help you create useful alerts.
  • This allows you to quickly create Cloud Monitoring alerts for a high rate of product catalog upload errors, a high rate of user event errors, high rate of prediction errors, high rate of search errors, and a reduction in the rate of user events being uploaded.
• Models: Status information for the most active Recommendations AI models.
  • Provides a status for active models indicating whether they are ready and the training schedule for the model.
• Analytics: Summary analytics data from the last 24 hours of event data.
  • Summary information for the effectiveness of the Retail services including total revenue from retail events, revenue from recommendations, average order value, average order value from recommendations, total searches, add-to-cart rate from searches, and the conversion rate from searches.
• Serving configs: Quick links for evaluating or deploying active serving configs.
  • This allows you to quickly tune, test, or learn how to deploy serving configs.

Task 4. Create and configure alerts for Retail services

In this task, you will configure an alert that fires when user events uploads are interrupted. This indicates that there is a major problem with the source of your user events, your retail site for example, that should be investigated.

Create a Retail alerting policy

You will now create a service-managed alerting policy using the Retail console.

1. In the GCP Console, on the Navigation menu (), click VIEW ALL PRODUCTS. In the Artificial Intelligence section, click Search for Retail > Monitoring to open the Retail Monitoring page.

2. Click Recommended Alerts.

3. Enable User events record reduction.

4. Click Get Code > Command Line.

This will display the gcloud command line and REST API syntax that you can use to create an alert that will fire if no Retail events have been uploaded for a period of 10 minutes. Note that this command does not include a notification channel, that will need to be specified if you want to use a command like this.

gcloud alpha monitoring policies create --policy=' { "displayName": "User events record reduction", "documentation": { "content": "No user events have been recorded for 10 minutes.", "mimeType": "text/markdown" }, "conditions": [ { "displayName": "user event record reduction", "conditionAbsent": { "filter": "metric.type=\"serviceruntime.googleapis.com/api/request_count\" resource.type=\"consumed_api\" resource.label.\"service\"=\"retail.googleapis.com\" resource.label.\"method\"=monitoring.regex.full_match(\"(google.cloud.retail.v2.UserEventService.CollectUserEvent|google.cloud.retail.v2.UserEventService.WriteUserEvent)\")", "duration": "600s", "trigger": { "count": 1 }, "aggregations": [ { "alignmentPeriod": "60s", "crossSeriesReducer": "REDUCE_SUM", "perSeriesAligner": "ALIGN_SUM" } ] } } ], "combiner": "OR", "enabled": true, "userLabels": { "retail": "event_record_reduction", "version": "1" }, "notificationChannels": [] }'

5. Click Close. You will use the console to set up the alert in the following steps.

6. In User events record reduction, click Notification Channels.

7. Click Manage Notification Channels.

This will open a new browser tab with the Cloud Monitoring Notification Channels page open.

8. Next to Emails, click Add New.

9. Enter your email and name details in the Create Email Channel dialog.

10. Click Save.

11. Close the Cloud Monitoring Notification Channels browser tab. This will return you to the Retail alert configuration tab.

12. Click the refresh icon next to Manage Notification Channels.

This will close the Manage Notifications Channels pop-up.

13. In User events record reduction, click Notification Channels again.

14. Select the Email notification channel for the email alert channel you have just created.

15. Click OK.

16. Click Submit.

Explore the alerting policy using the Cloud Operations Monitoring console

You will explore the information and controls available to you for Retail alerting policies in the Cloud Operations Monitoring console by opening the service-managed policy for user events record reduction that you created in the previous task.

1. In the GCP Console, on the Navigation menu (), click VIEW ALL PRODUCTS. In the Observability section, click Monitoring > Overview to open the Cloud Operations Monitoring overview page.

2. Click Alerting.

3. Under policies, click User events record reduction to open your newly created Retail alert.

• The overview chart will show you the most recent alerting history. You can change the chart from the default time range covering the last hour up to the last six weeks.
• The incidents sections allows you to view current active alerts or to select Show Closed Incidents to list all recent alerts, even those that have been responded to and closed.

4. Hover your mouse over the Edit button.

The button is greyed out and the mouse-over text indicates that you cannot edit a service-managed alerting policy.

Note that you can enable and disable the event on these page but you cannot change the alert parameters or change the notification channel.

5. On the Monitoring menu, click Monitoring Settings.

6. Click Notification channels.

Note that you can manage your notification channels here, adding or removing alerting paths as required. Any additional notification channels created here will be available for Retail service-managed policies you create in the Retail console.

Task 5. Review data ingestion and Cloud Logging logs for Retail

In this task, you will examine the data ingestion and other events in Cloud Logging.

1. In the GCP Console, on the Navigation menu (), click VIEW ALL PRODUCTS. In the Observability section, click Logging to open the Retail dashboard.

2. Click Logs Explorer.

3. Copy the following query string into the query edit dialog and click Run Query.

   jsonPayload.message=~"UNJOINED_WITH_CATALOG" jsonPayload.@type="type.googleapis.com/google.cloud.retail.logging.ErrorLog"

4. If no events are displayed, click Extend time by: 1 hour and extend the time by a few hours until you can see the user event import errors that were triggered when you imported user event data.

5. Click the > icon to the left of one of the events to expand it.

6. Click jsonPayload.importPayload to expand the import payload data.

The jsonPayload.importPayload.gcsPath string contains the name of the Cloud Storage source file that had unjoined event data. This is the recent_retail_events.json file you imported in an earlier lab.

7. Click status to expand it.

The status.message string contains the full error details, including the unmatched productId strings not found in the catalog. This data will help you identify the specific events that failed and allow you to identify the root cause of the problem.

8. Click Edit Query.

9. Copy the following query string into the query edit dialog and click Run Query.

   jsonPayload.serviceContext.service="retail.googleapis.com"

10. This will show you a wider range of error and more event data.

    • UNJOINED_WITH_CATALOG
    • MISSING_FIELD
    • INCORRECT_JSON_FORMAT

You can expand the jsonPayload.status property in these events to get the specific details that caused the event errors.

Task 6. Update product data in the catalog using the Retail API

You will now use curl to make calls to the Retail API to retrieve and update product catalog data.

Create an IAM service account to authenticate requests

1. Create an environment variable to store the Project ID:

   export PROJECT_ID=$(gcloud config get-value core/project)

2. Create an IAM service account for controlled access to the Retail API:

   export SA_NAME="retail-service-account" gcloud iam service-accounts create $SA_NAME --display-name $SA_NAME

Note: If you have created this service account in a previous lab this command will fail with an error stating that the account already exists. You can ignore that error and continue.

3. Bind the service account to the Retail Editor IAM role: gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:$SA_NAME@${PROJECT_ID}.iam.gserviceaccount.com" \ --role="roles/retail.editor"

If this succeeds then the updated project IAM policy, including this new binding, is displayed.

Allow the lab user account to use impersonation with the new service account

Creating a role binding on the service account for the lab user with the Service Account Token Creator role allows the lab user to use service account impersonation to safely generate limited duration authentication tokens for the service account. These tokens can then be used to interactively test access to APIs and services.

1. Create a role binding on the Retail API service account for your user account to permit impersonation:

   export USER_ACCOUNT=$(gcloud config list --format 'value(core.account)') gcloud iam service-accounts add-iam-policy-binding $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com --member "user:$USER_ACCOUNT" --role roles/iam.serviceAccountTokenCreator

2. Generate a temporary access token for the Retail API:

   export ACCESS_TOKEN=$(gcloud auth print-access-token --impersonate-service-account $SA_NAME@$PROJECT_ID.iam.gserviceaccount.com )

This command may fail as it can take up to 10 minutes for the Service Account Token Creator role to propagate. Retry this command after 1 minute if it fails, and retry until it succeeds. You will also see a warning informing you that the command is using impersonation. This is expected.

Get product data from the catalog using the Retail API

You will use the product URL in the catalog to retrieve the product data.

1. Store sample user event JSON data in an environment variable:

   export PRODUCT_ID="GGOEAAKQ137410" echo $PRODUCT_ID

2. Store the REST API URL for the product in your catalog using the Retail API in an environment variable:

   PRODUCT_URL="https://retail.googleapis.com/v2/projects/${PROJECT_ID}/locations/global/catalogs/default_catalog/branches/0/products/${PRODUCT_ID}" echo $PRODUCT_URL

This is the REST API URL for the product in the Retail API. This URL is used to manage the product in the catalog. Note that the URL includes bash environment variable substitutions for the Project ID and the Product_ID.

3. Retrieve the product data from the Catalog using the REST API using curl: curl -H "Authorization: Bearer ${ACCESS_TOKEN}" \ -H 'Content-Type: application/json' -X GET $PRODUCT_URL

The response returned will look like the following block of JSON.

{ "name": "projects/610724409905/locations/global/catalogs/default_catalog/branches/0/products/GGOEAAKQ137410", "id": "GGOEAAKQ137410", "type": "PRIMARY", "primaryProductId": "GGOEAAKQ137410", "categories": [ "Apparel" ], "title": "Android Iconic Sock", "priceInfo": { "currencyCode": "USD", "price": 17 }, "availability": "IN_STOCK" }

Note that the product title is Android Iconic Sock.

4. Define a JSON payload with an updated title. PRODUCT_DATA=" { 'title':'Android Iconic Sock', 'categories':['Apparel'] }" echo ${PRODUCT_DATA}

This update payload makes a minor change to the product title.

5. Update the catalog item using the REST API using curl to make a PATCH request: curl -H "Authorization: Bearer ${ACCESS_TOKEN}" \ -H 'Content-Type: application/json charset=utf-8' \ -X PATCH -d "${PRODUCT_DATA}" $PRODUCT_URL

This will return the updated product JSON in the response showing the updated product title.

{ "name": "projects/610724409905/locations/global/catalogs/default_catalog/branches/0/products/GGOEAAKQ137410", "id": "GGOEAAKQ137410", "type": "PRIMARY", "primaryProductId": "GGOEAAKQ137410", "categories": [ "Apparel" ], "title": "Android Iconic Sock", "availability": "IN_STOCK" }

Congratulations!

Congratulations, you've successfully explored the monitoring and management tools for the Retail API.