{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Lookout for Vision and Amazon A2I (Augmented AI) Integration with Model Retraining\n",
    "\n",
    "In this notebook, we will walk you through one path of how to use Amazon A2I Workteam results based from images that have returned inference results below a defined threshold from our Amazon Lookout for Vision model.\n",
    "\n",
    "We will be using the Amazon Lookout for Vision Python SDK. It gives you a programmatic way of interacting with this service and adds a lot of helper functions that complement the service. If you have not already installed the Amazon Lookout for Vision Python SDK, we will do that in an upcoming step.\n",
    "\n",
    "To help you learn about creating a model, Amazon Lookout for Vision provides example images of circuit boards (circuit_board) that you can use. These images are taken from https://docs.aws.amazon.com/lookout-for-vision/latest/developer-guide/su-prepare-example-images.html.\n",
    "\n",
    "In order to use this notebook successfully, make sure to clone the following repository: https://github.com/aws-samples/amazon-lookout-for-vision\n",
    "\n",
    "The workflow for this lab is as follows:\n",
    "* Define variables and install local dependencies\n",
    "* Create an Amazon Lookout for Vision model using the Amazon Lookout for Vision Python SDK\n",
    "* Create a human review Workteam or Workforce\n",
    "* Create a Human Task UI\n",
    "* Use an Amazon Lookout for Vision model to check images and then start a human loop based on inference results\n",
    "* Check the status of our Human Loops, waiting for the workers to complete open tasks\n",
    "* Retrain your Amazon Lookout for Vision model using the results from the human tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you have not cloned the repository yet, uncomment and execute the following:\n",
    "\n",
    "#!git clone https://github.com/aws-samples/amazon-lookout-for-vision.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisites and environmental variables\n",
    "\n",
    "First, let's install required libraries and define global variables.\n",
    "\n",
    "Libraries:\n",
    "* Amazon Lookout for Vision SDK\n",
    "\n",
    "Variables:\n",
    "* region: set this to the region where your project is located\n",
    "* project_name: this is the name of your Amazon Lookout for Vision project\n",
    "* bucket: provide the name of the S3 bucket where we will output the model results\n",
    "* a2i_output_path: provide the name of the S3 folder where A2I results will be stored\n",
    "* model_version: default setting is 1\n",
    "* workteam_arn: provide the name of the workforce or workteam ARN that you created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Install the Amazon Lookout for Vision SDK on the local notebook instance\n",
    "# Run \"pip list\" to viewed all currently installed libraries if needed\n",
    "\n",
    "#pip list\n",
    "\n",
    "!pip install lookoutvision\n",
    "!pip install simplejson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the following variables to match your Amazon Lookout for Vision project\n",
    "\n",
    "# Set the AWS region\n",
    "region = '<AWS REGION>'\n",
    "\n",
    "# Set your project name here\n",
    "project_name = '<CHANGE TO AMAZON LOOKOUT FOR VISION PROJECT NAME>'\n",
    "\n",
    "# Provide the name of the S3 bucket where we will output results and store images\n",
    "bucket = '<S3 BUCKET NAME>'\n",
    "\n",
    "# This will default to a value of 1; Since we're training a new model, leave this set to a value of 1\n",
    "model_version = '1'\n",
    "\n",
    "# Leave everything else in this cell as is; nothing else to modify!\n",
    "\n",
    "import os\n",
    "import boto3\n",
    "import botocore\n",
    "import io\n",
    "import uuid\n",
    "import time\n",
    "import simplejson as json\n",
    "import sagemaker\n",
    "import re\n",
    "import pprint\n",
    "\n",
    "from lookoutvision.metrics import Metrics\n",
    "from sagemaker import get_execution_role\n",
    "from lookoutvision.manifest import Manifest\n",
    "\n",
    "# setting up the s3 folder where a2i results will be stored\n",
    "a2i_results = f\"s3://{bucket}/a2i-results\"\n",
    "\n",
    "# Setting Role to the default SageMaker Execution Role\n",
    "role = get_execution_role()\n",
    "display(role)\n",
    "\n",
    "timestamp = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "# Amazon SageMaker client\n",
    "sagemaker_client = boto3.client('sagemaker', region)\n",
    "\n",
    "# Amazon Augment AI (A2I) client\n",
    "a2i = boto3.client('sagemaker-a2i-runtime')\n",
    "\n",
    "# Amazon Lookout for Vision client\n",
    "L4Vclient = boto3.client(\"lookoutvision\")\n",
    "\n",
    "# Amazon S3 client\n",
    "S3client = boto3.client('s3', region)\n",
    "\n",
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flowDefinitionName = 'fd-sagemaker-object-detection-demo-' + timestamp\n",
    "\n",
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "taskUIName = 'ui-sagemaker-object-detection-demo-' + timestamp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to add the necessary Amazon Lookout for Vision permissions to your SageMaker Execution Role shown above. Simply go to IAM in the console, select \"Roles\", locate the name of the role as shown above, and attach the \"AmazonLookoutVisionFullAccess\" and \"AmazonS3FullAccess\" policies. In a real deployment, we would only allow the minimum permissions needed to accomplish the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the S3 bucket if doesn't exist\n",
    "!aws s3 mb s3://{bucket}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMPORTANT: If you do not add a CORS configuration to the S3 buckets that contains your input data, human review tasks for those input data objects fail. See https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-permissions-security.html#a2i-cors-update for instructions.\n",
    "\n",
    "To set the required CORS headers on the S3 bucket that contains your input images in the S3 console, follow the directions detailed in https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-cors-configuration.html. Use the following CORS configuration code for the buckets that host your images. If you use the Amazon S3 console to add the policy to your bucket, you must use the JSON format.\n",
    "\n",
    "[{\n",
    "   \"AllowedHeaders\": [],\n",
    "   \"AllowedMethods\": [\"GET\"],\n",
    "   \"AllowedOrigins\": [\"*\"],\n",
    "   \"ExposeHeaders\": []\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Upload Circuit Board images to S3\n",
    "\n",
    "Now it's time to upload all the images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload images to S3 bucket:\n",
    "# Make sure you are in the correct directory to copy the circuitboard dataset or adjust the path accordingly\n",
    "\n",
    "!aws s3 cp ../circuitboard/train/normal s3://{bucket}/{project_name}/training/normal --recursive\n",
    "!aws s3 cp ../circuitboard/train/anomaly s3://{bucket}/{project_name}/training/anomaly --recursive\n",
    "\n",
    "!aws s3 cp ../circuitboard/test/normal s3://{bucket}/{project_name}/validation/normal --recursive\n",
    "!aws s3 cp ../circuitboard/test/anomaly s3://{bucket}/{project_name}/validation/anomaly --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Create an Amazon Lookout for Vision Project\n",
    "\n",
    "You have a couple of options on how to create your Amazon Lookout project (console, CLI or boto3). We chose boto3 SDK in this example. We highly recommend to check out the console, too.\n",
    "\n",
    "The steps we take with the SDK are:\n",
    "\n",
    "*Create a project (the name as been defined at the beginning)\n",
    "*Tell your project where to find your training dataset. This is done via the manifest file for training.\n",
    "*Tell your project where to find your test dataset. This is done via the manifest file for test.\n",
    "    *Note: This step is optional. In general all 'test' related code, etc. is optional. Amazon Lookout for Vision will also work with 'training' dataset only. We chose to use both as training and testing is a common (best) practice when training AI/ML models. And we should always let our customer know this to help them get to the next level.\n",
    "Create a model. This command will trigger the model training and validation.\n",
    "\n",
    "*Note: Training a model can take a few hours as it uses Deep Learning in the background. Once your model is trained, you can continue with this notebook to make predictions.\n",
    "\n",
    "Create a model. This command will trigger the model training and validation.\n",
    "Note: Training a model can (will) take a few hours as it uses Deep Learning in the background. Once your model is trained, you can continue with this notebook to make predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Create a manifest file from the dataset\n",
    "\n",
    "Now that we have our \"circuitboard\" images cloned to our local notebook at \"./circuitboard\", we need to generate a manifest file for training. Amazon Lookout for Vision uses this manifest file to determine the location of the files, as well as the labels associated with the files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create the manifest file\n",
    "\n",
    "from lookoutvision.manifest import Manifest\n",
    "mft = Manifest(\n",
    "    bucket=bucket,\n",
    "    s3_path=\"{}/\".format(project_name),\n",
    "    datasets=[\"training\", \"validation\"])\n",
    "mft_resp = mft.push_manifests()\n",
    "print(mft_resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Create an Amazon Lookout for Vision Project\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create an Amazon Lookout for Vision Project\n",
    "\n",
    "from lookoutvision.lookoutvision import LookoutForVision\n",
    "l4v = LookoutForVision(project_name=project_name)\n",
    "# If project does not exist: create it\n",
    "p = l4v.create_project()\n",
    "print(p)\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Create and train a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Create the training and test datasets from images in S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsets = l4v.create_datasets(mft_resp, wait=True)\n",
    "print(dsets)\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Fit a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l4v.fit(\n",
    "    output_bucket=bucket,\n",
    "    model_prefix=\"mymodel_\",\n",
    "    wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Wait for the model to finish training\n",
    "\n",
    "Training a model can (will) take a few hours as it uses Deep Learning in the background. Once your model is trained, you can continue with this notebook to make predictions. You can monitor the training progress by going to the Amazon Lookout for Vision Console and selecting the project, or by using \"describe_model\" below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 View the metrics of your model\n",
    "\n",
    "Let's check the metrics of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "met = Metrics(project_name=project_name)\n",
    "\n",
    "met.describe_model(model_version=model_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Host the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "l4v.deploy(\n",
    "    model_version=model_version,\n",
    "    wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Setup Amazon A2I (Augmented Reality)\n",
    "\n",
    "Now that we've successfully created a dataset and trained an Amazon Lookout for Vision model, we'll setup Amazon A2I.\n",
    "\n",
    "In this section, we will create a Workteam as well as a custom Human Task User Interface (UI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Creating human review Workteam or Workforce\n",
    "\n",
    "A workforce is the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce for human reviews. Whichever workforce type you choose, Amazon Augmented AI takes care of sending tasks to workers.\n",
    "\n",
    "When you use a private workforce, you also create work teams, a group of workers from your workforce that are assigned to Amazon Augmented AI human review tasks. You can have multiple work teams and can assign one or more work teams to each job.\n",
    "\n",
    "To create your Workteam, visit the instructions here: https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html\n",
    "\n",
    "After you have created your workteam, replace workteam_arn below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define workteam ARN\n",
    "\n",
    "workteam_arn = 'arn:aws:sagemaker:us-east-1:714751237672:workteam/private-crowd/lfv-a2i'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Create Human Task UI\n",
    "Create a human task UI resource, giving a UI template in liquid html. This template will be rendered to the human workers whenever human loop is required.\n",
    "\n",
    "For over 70 pre built UIs, check: https://github.com/aws-samples/amazon-a2i-sample-task-uis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In this section, we are creating a custom Human Task UI. When a human loop is created, workers will see this template.\n",
    "# For this notebook, the worker will simply need to select if the image is \"Normal\" or an \"Anomaly\".\n",
    "\n",
    "template = r\"\"\"\n",
    "<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "\n",
    "<crowd-form>\n",
    "  <crowd-image-classifier\n",
    "    name=\"crowd-image-classifier\"\n",
    "    src=\"{{ task.input.taskObject | grant_read_access }}\"\n",
    "    header=\"Please select the correct category for this image\"\n",
    "    categories=\"['Normal', 'Anomaly']\"\n",
    "  >\n",
    "    <full-instructions header=\"Classification Instructions\">\n",
    "      <p>Read the task carefully and inspect the image.</p>\n",
    "      <p>Choose the appropriate label that best suits the image.</p>\n",
    "    </full-instructions>\n",
    "\n",
    "    <short-instructions>\n",
    "      <p>Read the task carefully and inspect the image.</p>\n",
    "      <p>Choose the appropriate label that best suits the image.</p>\n",
    "    </short-instructions>\n",
    "  </crowd-image-classifier>\n",
    "</crowd-form>\n",
    "\"\"\"\n",
    "\n",
    "def create_task_ui():\n",
    "    '''\n",
    "    Creates a Human Task UI resource.\n",
    "\n",
    "    Returns:\n",
    "    struct: HumanTaskUiArn\n",
    "    '''\n",
    "    response = sagemaker_client.create_human_task_ui(\n",
    "        HumanTaskUiName=taskUIName,\n",
    "        UiTemplate={'Content': template})\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create task UI\n",
    "humanTaskUiResponse = create_task_ui()\n",
    "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n",
    "print(humanTaskUiArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Create the flow definition\n",
    "\n",
    "In this section, we're going to create a flow definition definition. Flow Definitions allow us to specify:\n",
    "\n",
    "The workforce that your tasks will be sent to.\n",
    "The instructions that your workforce will receive. This is called a worker task template.\n",
    "The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks.\n",
    "Where your output data will be stored.\n",
    "This demo is going to use the API, but you can optionally create this workflow definition in the console as well.\n",
    "\n",
    "For more details and instructions, see: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## create flow definition. this specifies the work team & human task UI arn\n",
    "\n",
    "create_workflow_definition_response = sagemaker_client.create_flow_definition(\n",
    "        FlowDefinitionName = flowDefinitionName,\n",
    "        RoleArn = role,\n",
    "        HumanLoopConfig = {\n",
    "            \"WorkteamArn\": workteam_arn,\n",
    "            \"HumanTaskUiArn\": humanTaskUiArn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"Select if the component is damaged or not.\",\n",
    "            \"TaskTitle\": \"Verify if the component is damaged or not\"\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : a2i_results\n",
    "        }\n",
    "    )\n",
    "flowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn'] # let's save this ARN for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe flow definition - status should be active before proceeding\n",
    "\n",
    "for x in range(60):\n",
    "    describeFlowDefinitionResponse = sagemaker_client.describe_flow_definition(FlowDefinitionName=flowDefinitionName)\n",
    "    print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n",
    "    if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 Make predictions and start a human loop based on inference results\n",
    "\n",
    "In the next section, we will loop through an array of new images, using the Amazon Lookout for Vision SDK to determine if our input images are damaged or not and if they're above or below a defined threshold; in this case, we're setting the threshold confidence at .70. If our result is below .70, we'll start a human loop for a worker to manually determine if our image is normal or an anomally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "from IPython.display import display\n",
    "\n",
    "# create an array of input images from local storage using the extra images from the dataset\n",
    "Incoming_Images_Dir = \"../circuitboard/extra_images\"\n",
    "Incoming_Images_Array = os.listdir(Incoming_Images_Dir)\n",
    "\n",
    "print(\"Checking \" + str(len(Incoming_Images_Array)) + \" images. Here we go!\")\n",
    "print(\"\\n\")\n",
    "\n",
    "human_loops_started = []\n",
    "\n",
    "SCORE_THRESHOLD = .70\n",
    "\n",
    "for fname in Incoming_Images_Array:\n",
    "    #Lookout for Vision inference using detect_anomalies\n",
    "    fname_full_path = (Incoming_Images_Dir + \"/\" + fname)\n",
    "    #display fname_full_path image\n",
    "    display(Image(width=\"400\",filename=fname_full_path))\n",
    "    with open(fname_full_path, \"rb\") as image:\n",
    "        modelresponse = L4Vclient.detect_anomalies(\n",
    "            ProjectName=project_name,\n",
    "            ContentType=\"image/jpeg\",  # or image/png for png format input image.\n",
    "            Body=image.read(),\n",
    "            ModelVersion=model_version,\n",
    "            )\n",
    "        modelresponseconfidence = (modelresponse[\"DetectAnomalyResult\"][\"Confidence\"])\n",
    "    print(\"Name of local file pulled from the array: \" + fname)\n",
    "    print(\"Is the PCB damaged? \" + str(modelresponse[\"DetectAnomalyResult\"][\"IsAnomalous\"]))\n",
    "    print(\"Confidence: \" + str(modelresponse[\"DetectAnomalyResult\"][\"Confidence\"]))\n",
    "    print(\"\\n\")\n",
    "    #End Lookout for Vision inference\n",
    "    if (modelresponseconfidence < SCORE_THRESHOLD):\n",
    "        #s3_fname='s3://%s/a2i-results/%s' % (bucket, fname)\n",
    "        s3_fname = (a2i_results + \"/\" + fname)\n",
    "        #copy local image to s3 for a2i if we're below the threshold so it can be seen in the human loop\n",
    "        print(\"Copying local image to S3 for A2I because confidence is below the threshold\")\n",
    "\n",
    "        !aws s3 cp {fname_full_path} {a2i_results}/\n",
    "            \n",
    "        humanLoopName = str(uuid.uuid4())\n",
    "        inputContent = {\n",
    "            \"initialValue\": modelresponseconfidence,\n",
    "            \"taskObject\": s3_fname\n",
    "        }\n",
    "        # start an a2i human review loop with an input\n",
    "        start_loop_response = a2i.start_human_loop(\n",
    "            HumanLoopName=humanLoopName,\n",
    "            FlowDefinitionArn=flowDefinitionArn,\n",
    "            HumanLoopInput={\n",
    "                \"InputContent\": json.dumps(inputContent)\n",
    "            }\n",
    "        )\n",
    "        human_loops_started.append(humanLoopName)\n",
    "        print(\"\\n\")\n",
    "        print(f'Object detection Confidence Score of %s is less than the threshold of %.2f' % (modelresponseconfidence, SCORE_THRESHOLD))\n",
    "        print(f'Starting human loop with name: {humanLoopName}  \\n')\n",
    "        print(\"-------------------------------------------------------------------------------------------\")\n",
    "    else:\n",
    "        print(f'Object detection Confidence Score of %s is above than the threshold of %.2f' % (modelresponseconfidence, SCORE_THRESHOLD))\n",
    "        print('No human loop created. \\n')\n",
    "        print(\"-------------------------------------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4.1 Check the status of the human loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Check the status of the human loops\n",
    "\n",
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4.2 Wait For Workers to Complete Task\n",
    "Since we are using a private workteam, we should go to the labeling UI to perform the inspection ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hint: use the output from the inference results above to help guide your answers for anomaly or normal!\n",
    "\n",
    "workteamName = workteam_arn[workteam_arn.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker_client.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4.3 Check the status of the human loop again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4.4 View the task results and move the taskobject to the correct folder for retraining "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once work is completed, Amazon A2I stores results in your S3 bucket and sends a Cloudwatch event. Your results should be available in the S3 a2i_output when all work is completed.\n",
    "\n",
    "Now that we have results from our human loops stored in s3, we can use that data to sort our images into the appropriate training folders and train a new model version!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# view the output of the human loop task and move the taskobject to the correct folder for retraining\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    split_string = re.split('s3://' +  bucket + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = split_string[1]\n",
    "    response = S3client.get_object(Bucket=bucket, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    pp.pprint(json_output)\n",
    "    print('\\n')\n",
    "    #Let's move the taskobject image to the correct folder for retraining now\n",
    "    humanAnswersResponse = (json_output[\"humanAnswers\"])\n",
    "    labelanswer = humanAnswersResponse[0][\"answerContent\"][\"crowd-image-classifier\"][\"label\"]\n",
    "    # let's also grab the taskObject value - we'll use this to move the object to the training folder in s3\n",
    "    inputContentResponse = (json_output[\"inputContent\"])\n",
    "    taskObjectResponse = (inputContentResponse[\"taskObject\"])\n",
    "    # view the results from the label\n",
    "    print(\"The results from this human loop show the image is \" + labelanswer)\n",
    "    print('\\n')\n",
    "    # move the image to the appropriate training folder\n",
    "    if (labelanswer == \"Normal\"):\n",
    "        # move object to the Normal training folder s3://a2i-lfv-output/image_folder/normal/\n",
    "        !aws s3 cp {taskObjectResponse} s3://{bucket}/{project_name}/training/normal/\n",
    "        print('\\n')\n",
    "        print(taskObjectResponse + \" has been moved to the Normal folder for training\")\n",
    "    else:\n",
    "        # move object to the Anomaly training folder\n",
    "        !aws s3 cp {taskObjectResponse} s3://{bucket}/{project_name}/training/anomaly/\n",
    "        print('\\n')\n",
    "        print(taskObjectResponse + \" has been moved to the Anomaly folder for training\")\n",
    "    print(\"------------------------------------------------------------------------------------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Training a new model from the modified dataset\n",
    "\n",
    "Training a new model version can be triggered as a batch job on a schedule, manually as needed, based on how many new images have been added to the training folders, etc.\n",
    "\n",
    "For this example, we will use the Lookout for Vision SDK to retrain our model using the images that we've now included in our modified dataset.\n",
    "\n",
    "https://github.com/awslabs/amazon-lookout-for-vision-python-sdk/blob/main/example/lookout_for_vision_example.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Generate a new manifest\n",
    "\n",
    "from lookoutvision.lookoutvision import LookoutForVision\n",
    "\n",
    "print(f'bucket = {bucket}, project name = {project_name}')\n",
    "mft = Manifest(\n",
    "    bucket=bucket,\n",
    "    s3_path=\"{}/\".format(project_name),\n",
    "    datasets=[\"training\", \"validation\"])\n",
    "print(f'mft = {mft}')\n",
    "mft_resp = mft.push_manifests()\n",
    "print (f'mft resp = {mft_resp}')\n",
    "dsets = l4v.update_datasets(mft_resp, wait=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model!\n",
    "\n",
    "l4v.fit(\n",
    "    output_bucket=bucket,\n",
    "    model_prefix=\"mymodelprefix_\",\n",
    "    wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View the metrics of the new model\n",
    "\n",
    "Now that we've trained a new model using newly added images, let's check the model metrics!\n",
    "We'll show the results from the first model and the second model this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All models of the same project\n",
    "met.describe_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop the model and cleanup resources\n",
    "\n",
    "Be sure to stop any hosted models, delete Jupyter notebooks that are no longer used, delete any Amazon Lookout for Vision projects you are no longer using, and remove objects from S3 to save costs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#If you are not using the model, stop to save costs! This can take up to 5 minutes.\n",
    "\n",
    "#change the model version to whichever model you're using within your current project\n",
    "model_version='1'\n",
    "l4v.stop_model(model_version=model_version)\n",
    "\n",
    "# print('Stopping model version ' + model_version  + ' for project ' + project_name )\n",
    "# response=L4Vclient.stop_model(ProjectName=project_name,\n",
    "#     ModelVersion=model_version)\n",
    "# print('Status: ' + response['Status'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}