{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contextual Bandits with Parametric Actions -- Experimentation Mode"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We demonstrate how you can use varying number of actions with contextual bandits algorithms in SageMaker. This notebook builds on \n",
    "the [Contextual Bandits example notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/bandits_statlog_vw_customEnv/bandits_statlog_vw_customEnv.ipynb) example notebook which used fixed number of actions. Please refer to that notebook for basics on contextual \n",
    "bandits. \n",
    "\n",
    "In the contextual bandit setting, an agent recommends an action given a state. This notebook introduces three features to bandit algorithms that make them applicable to a broader set of real-world problems. We use the movie recommendation problem as an example.\n",
    "1. The number of actions available to the agent can change over time. For example, the movies in the catalog changes over time.\n",
    "2. Each action may have features associated with it. For the movie recommendation problem, each movie can have features such as \n",
    "genre, cast, etc.\n",
    "3. The agent can produce a ranked list of actions/items. When recommending movies, it is natural that multiple movies are recommended at a time step.\n",
    "\n",
    "The contextual bandit agent will trade-off between exploitation and exploration to quickly learn user preferences and minimize \n",
    "poor recommendations. The bandit algorithms are appropriate to use in recommendation problems when there are many cold items (items which have no or little interaction data) in the catalog or if user preferences change over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Experimentation Mode?\n",
    "\n",
    "Contextual bandits are often used to train models by interacting with the real world. In movie recommendation, the bandit learns user preferences based on their feedback from past interactions. To test if bandit algorithms are applicable for your use case, you may want to test different algorithms and understand the impact of different features, hyper-parameters. Experimenting with real users can lead to poor experience due to unanticipated issues or poor performance. Experimenting in production comes with the complexity of working with infrastructure components (e.g. web services, data engines, databases) designed for scale. With Experimentation Mode, you can get started with a small dataset or a simulator and identify the algorithm, features and hyper-parameters that are best applicable for your use case. The experimentation is much faster, does not impact real users and easy to work with. Once you are satisfied with the algorithm performance, you can switch to [Deployment Mode](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/bandits_statlog_vw_customEnv/bandits_statlog_vw_customEnv.ipynb), where we provide infrastructure support that scales to production requirements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "    <img src=\"workflow.png\">\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-requisites \n",
    "\n",
    "### Imports\n",
    "\n",
    "To get started, we'll import the Python libraries we need, set up the environment with a few prerequisites for permissions and configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "import sys\n",
    "import os\n",
    "import json\n",
    "import glob\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import HTML\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "from misc import get_execution_role, wait_for_s3_object\n",
    "from sagemaker.rl import RLEstimator\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup S3 bucket\n",
    "\n",
    "Set up the linkage and authentication to the S3 bucket that you want to use for data and model outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage_session = sagemaker.session.Session()\n",
    "s3_bucket = sage_session.default_bucket()\n",
    "s3_output_path = \"s3://{}/\".format(s3_bucket)\n",
    "print(\"S3 bucket path: {}\".format(s3_output_path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check AWS Region\n",
    "aws_region = sage_session.boto_region_name\n",
    "if aws_region not in [\"us-west-2\"]:\n",
    "    raise Exception(\n",
    "        \"\"\"\n",
    "    This notebook can currently run only in us-west-2. Support for other regions\n",
    "    will be added soon.\n",
    "    \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure where training happens\n",
    "\n",
    "You can run this notebook on a `SageMaker notebook instance` or on your own machine. In both of these scenarios, you can do the training/inference in either the local or the SageMaker mode. The local mode uses the SageMaker Python SDK to run your code in a docker container locally. This can speed up iterative testing and debugging while using the same familiar Python SDK interface. You just need to set local_mode = True.  \n",
    "\n",
    "If local mode is `False`, then training/inference runs on a SageMaker machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run in local mode?\n",
    "local_mode = True\n",
    "\n",
    "if local_mode:\n",
    "    instance_type = \"local\"\n",
    "else:\n",
    "    instance_type = \"ml.c5.2xlarge\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an IAM role\n",
    "\n",
    "Either get the execution role when running from a SageMaker notebook instance `role = sagemaker.get_execution_role()` or, when running from local notebook instance, use utils method `role = get_execution_role()` to create an execution role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except:\n",
    "    role = get_execution_role()\n",
    "\n",
    "print(\"Using IAM role arn: {}\".format(role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulation environment (from MovieLens data)\n",
    "The simulation environment `src/env.py` simulates a live environment that can interact with the SageMaker bandits training to produce a recommender agent (or policy). The logic of reward generation resides in simulator itself. We simulate the online learning loop with feedback using this environment inside the training job itself. The simulator uses MovieLens 100k dataset.\n",
    "\n",
    "The training workflow is as follows:\n",
    "- **User sampling and candidate list generation**: The simulator picks a user u and a list of 100 items (defined by `item_pool_size`) at random, which is sent to the SageMaker agent for retrieving recommendations. This list consists of the movies that the user u has rated in the past, as we know the true user preferences (ratings) for these movies. In this simulator, we use `user_id` to identify the user and represent each movie using the genre features.\n",
    "- **Bandit Slate recommendation**: SageMaker bandit agent returns a recommendation - a list of top-k items.\n",
    "- **Feedback generation by simulating user behaviour**: The reward is given to the agent based on user ratings in the dataset. We assume a Cascade Click model, where the user scans the list top-down, and clicks on the item that she likes. We give a reward of 0 to all the items above the clicked item and a reward to 1 to the item that was clicked. No feedback is generated for the items below the clicked item.\n",
    "- **Feedback ingestion**: The corresponding rewards and the actions are reported to the agent for learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training job uses the files in the `src` folder. The descriptions of the important files are as follows:\n",
    "- `src/train.py` - This is the entrypoint for the training job. This file contains the main logic for training:\n",
    "    - It initializes a bandit agent.\n",
    "    - Starts an interaction loop in which the agent interacts with the envrionment, recommends some actions, ingests the feedback and improves over time.\n",
    "    - The agent is saved on S3 after training finishes and can be used later for inference.\n",
    "- `src/env.py` - This file implements the simulation environment using MovieLens 100K dataset. It also contains the logic for reward generation using the Cascade Click model.\n",
    "- `src/vw_agent.py` - This implements a bandit agent interface in python that communicates with a VW C++ process at the backend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### MovieLens 100K usage license\n",
    "Please be aware of the following requirements regarding acknowledgment, copyright and availability, cited from the [data set description page](http://files.grouplens.org/datasets/movielens/ml-100k-README.txt).\n",
    "\n",
    "The data set may be used for any research purposes under the following conditions:\n",
    "\n",
    " * The user may not state or imply any endorsement from the\n",
    "   University of Minnesota or the GroupLens Research Group.\n",
    " * The user must acknowledge the use of the data set in\n",
    "   publications resulting from the use of the data set\n",
    "   (see below for citation information).\n",
    " * The user may not redistribute the data without separate\n",
    "   permission.\n",
    " * The user may not use this information for any commercial or\n",
    "   revenue-bearing purposes without first obtaining permission\n",
    "   from a faculty member of the GroupLens Research Project at the\n",
    "   University of Minnesota.\n",
    "   \n",
    "If you have any further questions or comments, please contact GroupLens (grouplens-info@cs.umn.edu)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download MovieLens 100K and upload to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "curl -o ml-100k.zip http://files.grouplens.org/datasets/movielens/ml-100k.zip\n",
    "unzip ml-100k.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movielens_data_s3_path = sage_session.upload_data(\n",
    "    path=\"ml-100k\", bucket=s3_bucket, key_prefix=\"movielens/data\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Bandit model using the Python SDK Script mode\n",
    "\n",
    "If you are using local mode, the training will run on the notebook instance/your local machine. When using SageMaker for training, you can select a CPU instance. The RLEstimator is used for training the bandit agent.\n",
    "\n",
    "1. Specify the hyperparameters for the bandit algorithm and the environment configuration. \n",
    "2. Specify the source directory where the environment, training code and dependencies are present - `src` folder\n",
    "3. Specify the training entrypoint - `train.py`\n",
    "4. Specify the container image\n",
    "5. Define the training parameters such as the instance count, job name, S3 path for output and job name.  \n",
    "6. Specify the input dataset - `movielens_data_s3_path` in the `.fit` call"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the hyperparameters and the training job name prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = {\n",
    "    # Algorithm params\n",
    "    \"arm_features\": True,\n",
    "    \"exploration_policy\": \"regcbopt\",\n",
    "    \"mellowness\": 0.01,\n",
    "    # Env params\n",
    "    \"item_pool_size\": 100,\n",
    "    \"top_k\": 5,\n",
    "    \"total_interactions\": 2000,\n",
    "    \"max_users\": 100,\n",
    "}\n",
    "\n",
    "job_name_prefix = \"testbed-bandits-1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vw_image_uri = \"462105765813.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rl-vw-container:adf\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator = RLEstimator(\n",
    "    entry_point=\"train.py\",\n",
    "    source_dir=\"src\",\n",
    "    image_uri=vw_image_uri,\n",
    "    role=role,\n",
    "    instance_type=instance_type,\n",
    "    instance_count=1,\n",
    "    output_path=s3_output_path,\n",
    "    base_job_name=job_name_prefix,\n",
    "    hyperparameters=hyperparameters,\n",
    ")\n",
    "\n",
    "estimator.fit(inputs={\"movielens\": movielens_data_s3_path}, wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download the outputs to plot performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the training is complete, we can download the regrets data to plot the performance of the bandit agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = estimator.latest_training_job.job_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker saves the model in `model.tar.gz` and other job output in `output.tar.gz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if local_mode:\n",
    "    output_path_prefix = f\"{job_name}/output.tar.gz\"\n",
    "    model_path = f\"{job_name}/model.tar.gz\"\n",
    "\n",
    "else:\n",
    "    output_path_prefix = f\"{job_name}/output/output.tar.gz\"\n",
    "    model_path = f\"{job_name}/output/model.tar.gz\"\n",
    "\n",
    "sage_session.download_data(path=\"./output\", bucket=s3_bucket, key_prefix=output_path_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "tar -C ./output -xvzf ./output/output.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if local_mode:\n",
    "    output_path_local = \"output/data/output.json\"\n",
    "else:\n",
    "    output_path_local = \"output/output.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(output_path_local) as f:\n",
    "    all_regrets = json.load(f)\n",
    "\n",
    "all_regrets = {key: np.cumsum(val) for key, val in all_regrets.items()}\n",
    "df = pd.DataFrame(all_regrets)\n",
    "df.plot(title=\"Cumulative Regret\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regret at a time step is defined as the difference between the optimal reward that an agent can get and the actual reward that the agent got. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen from the above plot, the bandit agent leads to a much lesser cumulative regret compared to choosing the actions at random. If we run the training for a larger no. of interactions (`total_interactions`), we will observe that the cumulative regret curve flattens out, which means that the agent has learned the user preferences successfully and training has converged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see how we can use the trained model to perform inference on new data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a SageMaker model for inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage_session = sagemaker.local.LocalSession() if local_mode else sage_session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandit_model = sagemaker.model.Model(\n",
    "    image_uri=vw_image_uri,\n",
    "    role=role,\n",
    "    name=\"vw-model-1\",\n",
    "    model_data=f\"s3://{s3_bucket}/{model_path}\",\n",
    "    sagemaker_session=sage_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference on SageMaker can be performed using the following two modes:\n",
    "- **[Batch Transform](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html)**: Useful for scenarios that do not require a persistent serving endpoint with sub-second latency. As the name suggests, a batch transformation job processes a batch of data and is useful for achieving high throughput inference on large volumes of input data.\n",
    "- **Real-time inference**: This mode spins up a SageMaker HTTP web server end-point, that can serve predictions in real-time.\n",
    "\n",
    "We demonstrate both the modes in the cells below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Batch Transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup input data prefix and output data prefix for batch transform\n",
    "\n",
    "# S3 prefix where we will upload the test dataset\n",
    "batch_input = f\"s3://{s3_bucket}/{job_name}/batch_input/\"\n",
    "\n",
    "# S3 prefix where the batch transformation job will store the output\n",
    "batch_output = f\"s3://{s3_bucket}/{job_name}/batch_output/\"\n",
    "\n",
    "print(\"Input path for batch transform: {}\".format(batch_input))\n",
    "print(\"Output path for batch transform: {}\".format(batch_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create a transformer object that can launch batch transformation jobs\n",
    "\n",
    "batch_transformer = bandit_model.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type=instance_type,\n",
    "    output_path=batch_output,\n",
    "    assemble_with=\"Line\",\n",
    "    accept=\"application/jsonlines\",\n",
    "    max_payload=5,\n",
    "    max_concurrent_transforms=4,\n",
    "    strategy=\"MultiRecord\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating test dataset for inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we generate some test data instances using the MovieLens simulator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.env import MovieLens100KEnv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_k = 5\n",
    "env = MovieLens100KEnv(data_dir=\"./ml-100k\", item_pool_size=10, top_k=5, max_users=100)\n",
    "obs = env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"test_data_batch.jsonlines\", \"w\") as f:\n",
    "    for i in range(100):\n",
    "        user_features, items_features = obs\n",
    "        data_instance = {\n",
    "            \"shared_context\": user_features,\n",
    "            \"actions_context\": items_features.tolist(),\n",
    "            \"top_k\": 5,\n",
    "            \"user_id\": env.current_user_id,\n",
    "        }\n",
    "        f.write(json.dumps(data_instance))\n",
    "        f.write(\"\\n\")\n",
    "\n",
    "        # Step env with random actions to get next user and candidate items\n",
    "        obs, _, done, _ = env.step(actions=np.arange(top_k))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head -n2 test_data_batch.jsonlines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's upload this data to S3. Note that the format of the file should be `jsonlines`, which means each line of the file is a JSON dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage_session.upload_data(\n",
    "    path=\"test_data_batch.jsonlines\", bucket=s3_bucket, key_prefix=f\"{job_name}/batch_input\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_transformer.transform(\n",
    "    data=batch_input,\n",
    "    data_type=\"S3Prefix\",\n",
    "    content_type=\"application/jsonlines\",\n",
    "    split_type=\"Line\",\n",
    "    join_source=\"Input\",\n",
    ")\n",
    "batch_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download batch transform results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_transformer._current_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if local_mode:\n",
    "    batch_output_prefix = f\"{job_name}/batch_output/{batch_transformer._current_job_name}/\"\n",
    "else:\n",
    "    batch_output_prefix = f\"{job_name}/batch_output/\"\n",
    "\n",
    "sage_session.download_data(path=\"./output\", bucket=s3_bucket, key_prefix=batch_output_prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the head of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "head -n5 ./output/test_data_batch.jsonlines.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Real-time inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to do real-time inference, we can deploy the model behind a SageMaker endpoint and make requests as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bandit_model.deploy(initial_instance_count=1, instance_type=instance_type, endpoint_name=\"bandit\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = sagemaker.predictor.Predictor(\n",
    "    endpoint_name=\"bandit\",\n",
    "    sagemaker_session=bandit_model.sagemaker_session,\n",
    "    serializer=sagemaker.serializers.JSONSerializer(),\n",
    "    deserializer=sagemaker.deserializers.JSONDeserializer(),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.predict(\n",
    "    {\"shared_context\": None, \"actions_context\": [[0, 0, 1], [1, 0, 0], [1, 1, 1]], \"top_k\": 2}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean Up endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if \"predictor\" in locals():\n",
    "    predictor.delete_endpoint()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/reinforcement_learning|bandits_recsys_movielens_testbed|bandits_movielens_testbed.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}