{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retail Demo Store - Personalization Workshop - Lab 3\n",
    "\n",
    "In this lab we are going to build on the [prior lab](./Lab-2-Prepare-Personalize-and-import-data.ipynb) by creating Amazon Personalize domain recommenders and custom solutions for additional use cases.\n",
    "\n",
    "## Lab 3 Objectives\n",
    "\n",
    "In this lab we will accomplish the following steps.\n",
    "\n",
    "- Create retail domain recommenders for the following use cases:\n",
    "    - **Recommended For You**: will be used on the homepage (and other later workshops) to provide personalized recommendations for a given user. This recommender will be used for known users or warm anonymous users.\n",
    "    - **Popular Items by Views**: will be used on the homepage for new/cold anonymous users where no interaction history is available. Once a new/cold user has a few interactions (product views), the homepage will switch to the Recommended For You recommender.\n",
    "- Create custom solutions and solution versions for the following use cases:\n",
    "    - **Similar Items**: will be used on the product detail and \"live\" pages to display similar items to a given item.\n",
    "    - **Personalized-Ranking**: will be used to rerank featured products and search results.\n",
    "    - **Item Attribute Affinity**: user segmentation model that will be used in a later lab.\n",
    "    \n",
    "This lab should take about 70-90 minutes to complete. However, most of the time will be waiting for model training jobs to complete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Just as in the previous labs, we have to prepare our environment by importing dependencies and creating clients.\n",
    "\n",
    "### Import dependencies\n",
    "\n",
    "The following libraries are needed for this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "import time\n",
    "from botocore.exceptions import ClientError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create clients\n",
    "\n",
    "We will need the following AWS service clients in this lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "personalize = boto3.client('personalize')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load variables saved in prior labs\n",
    "\n",
    "At the end of Lab 1 we saved some variables that we'll need in this lab. The following cell will load those variables into this lab environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Recommenders\n",
    "\n",
    "With our three datasets imported into our dataset group, we can now turn to creating recommenders and solutions. We'll start with pre-configured recommenders that match some of our core use cases.\n",
    "\n",
    "Let's start by listing the recipes for the `ECOMMERCE` domain. From these recipes we will be creating the following three recommenders.\n",
    "\n",
    "- **Recommended For You** - will be used on the home page for the \"Inspired by your shopping trends\" grid.\n",
    "- **Popular Items By Views** - will be used on the home page for new/cold users with no interactions so we can bootstrap the user experience with popular products. Once the user is \"warm\", we will switch to the Recommended For You recommender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize.list_recipes(domain = \"ECOMMERCE\")\n",
    "print(json.dumps(response['recipes'], indent=2, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Recommended For You recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    response = personalize.create_recommender(\n",
    "      name = 'retaildemostore-recommended-for-you',\n",
    "      recipeArn = 'arn:aws:personalize:::recipe/aws-ecomm-recommended-for-you',\n",
    "      datasetGroupArn = dataset_group_arn\n",
    "    )\n",
    "    rfy_recommender_arn = response['recommenderArn']\n",
    "    print(json.dumps(response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this recommender, seemingly')\n",
    "    paginator = personalize.get_paginator('list_recommenders')\n",
    "    for paginate_result in paginator.paginate(datasetGroupArn = dataset_group_arn):\n",
    "        for recommender in paginate_result['recommenders']:\n",
    "            if recommender['name'] == 'retaildemostore-recommended-for-you':\n",
    "                rfy_recommender_arn = recommender['recommenderArn']\n",
    "                break\n",
    "                \n",
    "print(f'Recommended For You recommender ARN = {rfy_recommender_arn}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Popular Items By Views recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    response = personalize.create_recommender(\n",
    "      name = 'retaildemostore-popular-items',\n",
    "      recipeArn = 'arn:aws:personalize:::recipe/aws-ecomm-popular-items-by-views',\n",
    "      datasetGroupArn = dataset_group_arn\n",
    "    )\n",
    "    most_viewed_recommender_arn = response['recommenderArn']\n",
    "    print(json.dumps(response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this recommender, seemingly')\n",
    "    paginator = personalize.get_paginator('list_recommenders')\n",
    "    for paginate_result in paginator.paginate(datasetGroupArn = dataset_group_arn):\n",
    "        for recommender in paginate_result['recommenders']:\n",
    "            if recommender['name'] == 'retaildemostore-popular-items':\n",
    "                most_viewed_recommender_arn = recommender['recommenderArn']\n",
    "                break\n",
    "                \n",
    "print(f'Most Viewed recommender ARN = {most_viewed_recommender_arn}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Custom Solutions\n",
    "\n",
    "The recommenders created above do not cover all of the personalization use cases that we want to implement in the Retail Demo Store. We also want to provide related items recommendations on the product detail page, personalize the order of featured products displayed on the home page, and we want to personalize the search results returned from Open Search in the search widget. To implement these use cases, we will create a custom solution using the [Similar-Items](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-similar-items.html) and [Personalized-Ranking](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-search.html) recipes.\n",
    "\n",
    "In addition, we also want to use Personalize to create user segments based on user affinity with specific product attributes. We'll explore this is a later lab but let's go ahead and create a custom solution using the [Item-Attribute-Affinity](https://docs.aws.amazon.com/personalize/latest/dg/item-attribute-affinity-recipe.html) recipe.\n",
    "\n",
    "These custom solutions will use the same datasets that we already implemented so all we need to do is create a solution and solution version for each recipe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List Recipes\n",
    "\n",
    "First, let's list all available recipes that aren't associated with a domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize.list_recipes()\n",
    "custom_recipes = []\n",
    "for recipe in response['recipes']:\n",
    "    if not recipe.get('domain'):\n",
    "        custom_recipes.append(recipe)\n",
    "        \n",
    "print(json.dumps(custom_recipes, indent=2, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, there are several recipes to choose from. Let's declare the recipes for the two custom solutions we will have to create."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Personalize Recipe for Similar Items\n",
    "\n",
    "In use-cases where we have an item/product and we want to display similar items based on the co-interactions of all users as well as draw upon thematic similarities based on item metadata, we can use the [Similar-Items](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-similar-items.html) recipe to provide related items recommendations.\n",
    "\n",
    "> The Similar-Items (aws-similar-items) generates recommendations for items that are similar to an item you specify. Use Similar-Items to help customers discover new items in your catalog based on their previous behavior and item metadata. Recommending similar items can increase user engagement, click-through rate, and conversion rate for your application.\n",
    "\n",
    "> Similar-Items calculates similarity based on interactions data and any item metadata you provide. It takes into account the co-occurrence of the item in user histories in your Interaction dataset, and any item metadata similarities. For example, with Similar-Items Amazon Personalize could recommend items customers frequently bought together with a similar style (Categorical metadata), or movies that different users also watched with a similar description (Unstructured text metadata).\n",
    "\n",
    "Note that Personalize also has the [SIMS](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-sims.html) recipe for the related items use case. However, SIMS only trains on co-interaction data (i.e. the interactions dataset) and does not consider item metadata. Since we may have some items with fewer (or no) interactions, the Similar-Items recipe is a better match for the Retail Demo Store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_items_recipe_arn = \"arn:aws:personalize:::recipe/aws-similar-items\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Personalize Recipe for Personalized Ranking\n",
    "\n",
    "In use-cases where we have a curated list of products, we can use the [Personalized-Ranking](https://docs.aws.amazon.com/personalize/latest/dg/native-recipe-search.html) recipe to reorder the products for the current user.\n",
    "\n",
    "> The Personalized-Ranking recipe generates personalized rankings. A personalized ranking is a list of recommended items that are re-ranked for a specific user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranking_recipe_arn = \"arn:aws:personalize:::recipe/aws-personalized-ranking\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare Personalize Recipe for Item Attribute Affinity\n",
    "\n",
    "For the user segmentation use case, we will generate groups of users (segments) with an affinity for specific item attributes. We will explore this use case further in an upcoming lab but for now we will create a custom solution and solution version using the [Item-Attribute-Affinity](https://docs.aws.amazon.com/personalize/latest/dg/item-attribute-affinity-recipe.html) recipe.\n",
    "\n",
    "> The Item-Attribute-Affinity (aws-item-attribute-affinity) recipe is a USER_SEGMENTATION recipe that creates a user segment (group of users) for each item attribute that you specify. Use Item-Attribute-Affinity to learn more about your users and take actions based on their respective user segments.\n",
    "\n",
    "> For example, you might want to create a marketing campaign for your retail application based on user preferences for shoe types in your catalog. Item-Attribute-Affinity would create a user segment for each shoe type based data in your Interactions and Items datasets. You could use this to promote different shoes to different user segments based on the likelihood that they will take an action (for example, click a shoe or purchase a shoe). Other uses might include promoting different movie genres to different users or identifying prospective job applicant based on job type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_attribute_affinity_recipe_arn = 'arn:aws:personalize:::recipe/aws-item-attribute-affinity'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Custom Solutions and Solution Versions\n",
    "\n",
    "With our recipes defined, we can now create our solutions and solution versions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Similar Items Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similar_items_solution_version_arn = None\n",
    "\n",
    "try:\n",
    "    create_solution_response = personalize.create_solution(\n",
    "        name = \"retaildemostore-related-items\",\n",
    "        datasetGroupArn = dataset_group_arn,\n",
    "        recipeArn = similar_items_recipe_arn\n",
    "    )\n",
    "\n",
    "    similar_items_solution_arn = create_solution_response['solutionArn']\n",
    "    print(json.dumps(create_solution_response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this solution, seemingly')\n",
    "    paginator = personalize.get_paginator('list_solutions')\n",
    "    for paginate_result in paginator.paginate(datasetGroupArn = dataset_group_arn):\n",
    "        for solution in paginate_result['solutions']:\n",
    "            if solution['name'] == 'retaildemostore-related-items':\n",
    "                similar_items_solution_arn = solution['solutionArn']\n",
    "                print(f'Similar Items solution ARN = {similar_items_solution_arn}')\n",
    "                \n",
    "                response = personalize.list_solution_versions(\n",
    "                    solutionArn = similar_items_solution_arn,\n",
    "                    maxResults = 100\n",
    "                )\n",
    "                if len(response['solutionVersions']) > 0:\n",
    "                    similar_items_solution_version_arn = response['solutionVersions'][-1]['solutionVersionArn']\n",
    "                    print(f'Will use most recent solution version for this solution: {similar_items_solution_version_arn}')\n",
    "                    \n",
    "                break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Similar Items Solution Version\n",
    "\n",
    "Next we can create a solution version for the solution. This is where the model is trained for this custom solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not similar_items_solution_version_arn:\n",
    "    create_solution_version_response = personalize.create_solution_version(\n",
    "        solutionArn = similar_items_solution_arn\n",
    "    )\n",
    "\n",
    "    similar_items_solution_version_arn = create_solution_version_response['solutionVersionArn']\n",
    "    print(json.dumps(create_solution_version_response, indent=2))\n",
    "else:\n",
    "    print(f'Solution version {similar_items_solution_version_arn} already exists; not creating')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Personalized Ranking Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ranking_solution_version_arn = None\n",
    "\n",
    "try:\n",
    "    create_solution_response = personalize.create_solution(\n",
    "        name = \"retaildemostore-personalized-ranking\",\n",
    "        datasetGroupArn = dataset_group_arn,\n",
    "        recipeArn = ranking_recipe_arn\n",
    "    )\n",
    "\n",
    "    ranking_solution_arn = create_solution_response['solutionArn']\n",
    "    print(json.dumps(create_solution_response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this solution, seemingly')\n",
    "    paginator = personalize.get_paginator('list_solutions')\n",
    "    for paginate_result in paginator.paginate(datasetGroupArn = dataset_group_arn):\n",
    "        for solution in paginate_result['solutions']:\n",
    "            if solution['name'] == 'retaildemostore-personalized-ranking':\n",
    "                ranking_solution_arn = solution['solutionArn']\n",
    "                print(f'Ranking solution ARN = {ranking_solution_arn}')\n",
    "                \n",
    "                response = personalize.list_solution_versions(\n",
    "                    solutionArn = ranking_solution_arn,\n",
    "                    maxResults = 100\n",
    "                )\n",
    "                if len(response['solutionVersions']) > 0:\n",
    "                    ranking_solution_version_arn = response['solutionVersions'][-1]['solutionVersionArn']\n",
    "                    print(f'Will use most recent solution version for this solution: {ranking_solution_version_arn}')\n",
    "                    \n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Personalized Ranking Solution Version\n",
    "\n",
    "Next we can create a solution version for the solution. This is where the model is trained for this custom solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not ranking_solution_version_arn:\n",
    "    create_solution_version_response = personalize.create_solution_version(\n",
    "        solutionArn = ranking_solution_arn\n",
    "    )\n",
    "\n",
    "    ranking_solution_version_arn = create_solution_version_response['solutionVersionArn']\n",
    "    print(json.dumps(create_solution_version_response, indent=2))\n",
    "else:\n",
    "    print(f'Solution version {ranking_solution_version_arn} already exists; not creating')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Item Attribute Affinity Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_attribute_affinity_solution_version_arn = None\n",
    "\n",
    "try:\n",
    "    create_solution_response = personalize.create_solution(\n",
    "        name = \"retaildemostore-item-attribute-affinity\",\n",
    "        datasetGroupArn = dataset_group_arn,\n",
    "        recipeArn = item_attribute_affinity_recipe_arn\n",
    "    )\n",
    "\n",
    "    item_attribute_affinity_solution_arn = create_solution_response['solutionArn']\n",
    "    print(json.dumps(create_solution_response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this solution, seemingly')\n",
    "    paginator = personalize.get_paginator('list_solutions')\n",
    "    for paginate_result in paginator.paginate(datasetGroupArn = dataset_group_arn):\n",
    "        for solution in paginate_result['solutions']:\n",
    "            if solution['name'] == 'retaildemostore-item-attribute-affinity':\n",
    "                item_attribute_affinity_solution_arn = solution['solutionArn']\n",
    "                print(f'Item Attribute Affinity solution ARN = {item_attribute_affinity_solution_arn}')\n",
    "                \n",
    "                response = personalize.list_solution_versions(\n",
    "                    solutionArn = item_attribute_affinity_solution_arn,\n",
    "                    maxResults = 100\n",
    "                )\n",
    "                if len(response['solutionVersions']) > 0:\n",
    "                    item_attribute_affinity_solution_version_arn = response['solutionVersions'][-1]['solutionVersionArn']\n",
    "                    print(f'Will use most recent solution version for this solution: {item_attribute_affinity_solution_version_arn}')\n",
    "                    \n",
    "                break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not item_attribute_affinity_solution_version_arn:\n",
    "    create_solution_version_response = personalize.create_solution_version(\n",
    "        solutionArn = item_attribute_affinity_solution_arn\n",
    "    )\n",
    "\n",
    "    item_attribute_affinity_solution_version_arn = create_solution_version_response['solutionVersionArn']\n",
    "    print(json.dumps(create_solution_version_response, indent=2))\n",
    "else:\n",
    "    print(f'Solution version {item_attribute_affinity_solution_version_arn} already exists; not creating')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for Recommenders and Solution Versions to Complete\n",
    "\n",
    "It can take 40-60 minutes for all recommenders and solution versions to be created. During this process a model is being trained and tested with the data contained within your datasets. The duration of training jobs can increase based on the size of the dataset, training parameters and a selected recipe. We submitted requests for all three recommenders and two custom solutions and versions at once so they are trained in parallel. In the cells below we will wait for all recommenders and solution versions to finish.\n",
    "\n",
    "While you are waiting for this process to complete you can learn more about [recommenders](https://docs.aws.amazon.com/personalize/latest/dg/creating-recommenders.html) and [custom solutions](https://docs.aws.amazon.com/personalize/latest/dg/training-deploying-solutions.html).\n",
    "\n",
    "The following cell waits for all recommenders to become active."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "recommender_arns = [ rfy_recommender_arn, most_viewed_recommender_arn ]\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    for recommender_arn in reversed(recommender_arns):\n",
    "        response = personalize.describe_recommender(\n",
    "            recommenderArn = recommender_arn\n",
    "        )\n",
    "        status = response[\"recommender\"][\"status\"]\n",
    "\n",
    "        if status == \"ACTIVE\":\n",
    "            print(f'Recommender {recommender_arn} successfully completed')\n",
    "            recommender_arns.remove(recommender_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(f'Recommender {recommender_arn} failed')\n",
    "            if response[\"recommender\"].get('failureReason'):\n",
    "                print('   Reason: ' + response[\"recommender\"]['failureReason'])\n",
    "            recommender_arns.remove(recommender_arn)\n",
    "\n",
    "    if len(recommender_arns) > 0:\n",
    "        print('At least one recommender is still in progress')\n",
    "        time.sleep(60)\n",
    "    else:\n",
    "        print(\"All recommenders have completed\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for custom solution versions to become active\n",
    "\n",
    "The following cell waits for the solution versions for the similar items, personalized ranking, and item attribute affinity use cases to become active. It's likely that they're already active (or close to being active) since they were being created in parallel with the recommenders. Nevertheless, we'll make sure they are active too before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "soln_ver_arns = [ \n",
    "    similar_items_solution_version_arn, \n",
    "    ranking_solution_version_arn, \n",
    "    item_attribute_affinity_solution_version_arn \n",
    "]\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    for soln_ver_arn in reversed(soln_ver_arns):\n",
    "        soln_ver_response = personalize.describe_solution_version(\n",
    "            solutionVersionArn = soln_ver_arn\n",
    "        )\n",
    "        status = soln_ver_response[\"solutionVersion\"][\"status\"]\n",
    "\n",
    "        if status == \"ACTIVE\":\n",
    "            print(f'Solution version {soln_ver_arn} successfully completed')\n",
    "            soln_ver_arns.remove(soln_ver_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(f'Solution version {soln_ver_arn} failed')\n",
    "            if soln_ver_response[\"solutionVersion\"].get('failureReason'):\n",
    "                print('   Reason: ' + soln_ver_response[\"solutionVersion\"]['failureReason'])\n",
    "            soln_ver_arns.remove(soln_ver_arn)\n",
    "\n",
    "    if len(soln_ver_arns) > 0:\n",
    "        print('At least one solution version is still in progress')\n",
    "        time.sleep(60)\n",
    "    else:\n",
    "        print(\"All solution versions have completed\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate Offline Metrics for Recommenders and Custom Solution Versions\n",
    "\n",
    "Amazon Personalize provides [offline metrics](https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html#working-with-training-metrics-metrics) for recommenders and custom solutions that allow you to evaluate the accuracy of the model before you deploy the model in your application. Metrics can also be used to view the effects of modifying a custom solution's hyperparameters or to compare the metrics between solutions that use the same training data but created with different recipes.\n",
    "\n",
    "Let's retrieve the metrics for the recommenders and custom solution versions we just created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Recommended For You Recommender Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize.describe_recommender(\n",
    "    recommenderArn = rfy_recommender_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(response['recommender']['modelMetrics'], indent=2, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Popular Viewed Recommender Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = personalize.describe_recommender(\n",
    "    recommenderArn = most_viewed_recommender_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(response['recommender']['modelMetrics'], indent=2, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Similar Items Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = similar_items_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(get_solution_metrics_response['metrics'], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Personalized Ranking Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = ranking_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(get_solution_metrics_response['metrics'], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Item Attribute Affinity Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = item_attribute_affinity_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(get_solution_metrics_response['metrics'], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can learn more about interpreting offline metrics in the Personalize [documentation](https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html) and in the blog post on [A/B testing](https://aws.amazon.com/blogs/machine-learning/using-a-b-testing-to-measure-the-efficacy-of-recommendations-generated-by-amazon-personalize/) with Personalize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create campaigns for Similar Items and Personalized Ranking solutions\n",
    "\n",
    "Once we're satisfied with our solution versions, we need to create campaigns for the custom solution versions created for the similar items and personalized ranking recipes. This is required so we have an real-time API endpoints that can be called from the Recommendations microservice to return related items for the product detail page and rerank products for the featured products and search widgets. When creating a campaign you can specify the minimum transactions per second (`minProvisionedTPS`) that you expect to make against the service for this campaign. Personalize will automatically scale resources for the inference endpoint up and down for the campaign to match demand but will never scale below `minProvisionedTPS`.\n",
    "\n",
    "Let's create campaigns for the similar items and personalized ranking custom solution versions with each set at `minProvisionedTPS` of 1 (which is also the default if not specified).\n",
    "\n",
    "Note: For the item attribute affinity custom solution, we don't need a campaign since it only supports generating user segments using batch segmentation jobs. We'll explore this in a later lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Similar Items campaign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    create_campaign_response = personalize.create_campaign(\n",
    "        name = \"retaildemostore-related-items\",\n",
    "        solutionVersionArn = similar_items_solution_version_arn,\n",
    "        minProvisionedTPS = 1\n",
    "    )\n",
    "\n",
    "    similar_items_campaign_arn = create_campaign_response['campaignArn']\n",
    "    print(json.dumps(create_campaign_response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this campaign, seemingly. Will update campaign instead.')\n",
    "    paginator = personalize.get_paginator('list_campaigns')\n",
    "    for paginate_result in paginator.paginate(solutionArn = similar_items_solution_arn):\n",
    "        for campaign in paginate_result['campaigns']:\n",
    "            if campaign['name'] == 'retaildemostore-related-items':\n",
    "                similar_items_campaign_arn = campaign['campaignArn']\n",
    "                print(f'Found existing campaign for solution: {similar_items_campaign_arn}')\n",
    "                \n",
    "                response = personalize.describe_campaign(campaignArn = similar_items_campaign_arn)\n",
    "                if response['campaign']['solutionVersionArn'] == similar_items_solution_version_arn:\n",
    "                    print('Campaign is already using the latest solution version')\n",
    "                else:\n",
    "                    print('Updating campaign with the latest solution version')\n",
    "                    response = personalize.update_campaign(\n",
    "                        campaignArn = similar_items_campaign_arn,\n",
    "                        solutionVersionArn = similar_items_solution_version_arn,\n",
    "                        minProvisionedTPS = 1\n",
    "                    )\n",
    "                    print(json.dumps(response, indent=2))\n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Personalized Ranking campaign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    create_campaign_response = personalize.create_campaign(\n",
    "        name = \"retaildemostore-personalized-ranking\",\n",
    "        solutionVersionArn = ranking_solution_version_arn,\n",
    "        minProvisionedTPS = 1\n",
    "    )\n",
    "\n",
    "    ranking_campaign_arn = create_campaign_response['campaignArn']\n",
    "    print(json.dumps(create_campaign_response, indent=2))\n",
    "except personalize.exceptions.ResourceAlreadyExistsException:\n",
    "    print('You aready created this campaign, seemingly. Will update campaign instead.')\n",
    "    paginator = personalize.get_paginator('list_campaigns')\n",
    "    for paginate_result in paginator.paginate(solutionArn = ranking_solution_arn):\n",
    "        for campaign in paginate_result['campaigns']:\n",
    "            if campaign['name'] == 'retaildemostore-personalized-ranking':\n",
    "                ranking_campaign_arn = campaign['campaignArn']\n",
    "                print(f'Found existing campaign for solution: {ranking_campaign_arn}')\n",
    "                \n",
    "                response = personalize.describe_campaign(campaignArn = ranking_campaign_arn)\n",
    "                if response['campaign']['solutionVersionArn'] == ranking_solution_version_arn:\n",
    "                    print('Campaign is already using the latest solution version')\n",
    "                else:\n",
    "                    print('Updating campaign with the latest solution version')\n",
    "                    response = personalize.update_campaign(\n",
    "                        campaignArn = ranking_campaign_arn,\n",
    "                        solutionVersionArn = ranking_solution_version_arn,\n",
    "                        minProvisionedTPS = 1\n",
    "                    )\n",
    "                    print(json.dumps(response, indent=2))\n",
    "                break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for campaigns to Have ACTIVE Status\n",
    "\n",
    "It can take 15-20 minutes for a campaign to be fully created. \n",
    "\n",
    "While you are waiting for this to complete you can learn more about campaigns here: https://docs.aws.amazon.com/personalize/latest/dg/campaigns.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "campaign_arns = [ similar_items_campaign_arn, ranking_campaign_arn ]\n",
    "\n",
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    for campaign_arn in reversed(campaign_arns):\n",
    "        campaign_response = personalize.describe_campaign(\n",
    "            campaignArn = campaign_arn\n",
    "        )\n",
    "        status = campaign_response[\"campaign\"][\"status\"]\n",
    "        if status == 'ACTIVE' and campaign_response.get('latestCampaignUpdate'):\n",
    "            status = campaign_response['latestCampaignUpdate']['status']\n",
    "\n",
    "        if status == \"ACTIVE\":\n",
    "            print(f'Campaign {campaign_arn} successfully completed')\n",
    "            campaign_arns.remove(campaign_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(f'Campaign {campaign_arn} failed')\n",
    "            if campaign_response[\"campaign\"].get('failureReason'):\n",
    "                print('   Reason: ' + campaign_response[\"campaign\"]['failureReason'])\n",
    "            campaign_arns.remove(campaign_arn)\n",
    "\n",
    "    if len(campaign_arns) > 0:\n",
    "        print('At least one campaign is still in progress')\n",
    "        time.sleep(60)\n",
    "    else:\n",
    "        print(\"All campaigns have completed\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 3 Summary - What have we accomplished?\n",
    "\n",
    "In this lab we created retail domain recommenders for different personalization use cases. We also created custom solutions for similar items and personalized ranking as well as item attribute affinity (user segmentation) use cases.\n",
    "\n",
    "In the next lab we will examine the recommendations provided by the recommenders and the custom campaigns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Store variables needed in the next lab\n",
    "\n",
    "We will pass some variables initialized in this lab by storing them in the notebook environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store recommender ARNs\n",
    "%store rfy_recommender_arn\n",
    "%store most_viewed_recommender_arn\n",
    "\n",
    "# Store solution version ARNs\n",
    "%store similar_items_solution_version_arn\n",
    "%store ranking_solution_version_arn\n",
    "%store item_attribute_affinity_solution_version_arn\n",
    "\n",
    "# Store campaign ARNs\n",
    "%store similar_items_campaign_arn\n",
    "%store ranking_campaign_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continue to [Lab 4](./Lab-4-Evaluate-recommendations.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}