{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Image Moderation with Human-in-the-loop\n", "\n", "Amazon Augmented AI (Amazon A2I) makes it easy to build the workflows required for human review of ML predictions. Amazon A2I brings human review to all developers, removing the undifferentiated heavy lifting associated with building human review systems or managing large numbers of human reviewers. \n", "\n", "Amazon A2I provides built-in human review workflows for common machine learning use cases, such as content moderation and text extraction from documents, which allows predictions from Amazon Rekognition and Amazon Textract to be reviewed easily. You can also create your own workflows for ML models built on Amazon SageMaker or any other tools. Using Amazon A2I, you can allow human reviewers to step in when a model is unable to make a high confidence prediction or to audit its predictions on an on-going basis. Learn more [here](https://aws.amazon.com/augmented-ai/).\n", "\n", "In this tutorial, we will show how you can use Amazon A2I directly within your API calls to Rekognition's Image Moderation API. \n", "\n", "For more in depth instructions, visit [here](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-getting-started.html).\n", "\n", "![image-moderation-hitl-arc](../images/image-moderation-hitl-architecture.png)\n", "\n", "- [Step 1: Setup Notebook](#step1)\n", "- [Step 2: Moderate image using Rekognition image moderation API](#step2)\n", "- [Step 3: Set up A2I for human review](#step3)\n", " - [Step 3.1: Create Work Team](#step31)\n", " - [Step 3.2: Create Human Task UI](#step32)\n", " - [Step 3.3: Creating the Flow Definition](#step33)\n", "- [Step 4: Moderate image with human-in-the-loop](#step4)\n", "- [Step 5: Check A2I generated result in S3](#step5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Step 1: Setup Notebook \n", "\n", "Run the below cell to install/update Python dependencies if you run the lab using a local IDE. It is optional if you use a SageMaker Studio Juypter Notebook, which already includes the dependencies in the kernel. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# First, let's get the latest installations of our dependencies\n", "%pip install -qU pip\n", "%pip install boto3 -qU\n", "%pip install IPython -qU" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import sagemaker as sm\n", "import os\n", "import datetime\n", "from IPython.display import Image as IImage, display\n", "from IPython.display import HTML, display\n", "import uuid\n", "import json\n", "\n", "# variables\n", "data_bucket = sm.Session().default_bucket()\n", "region = boto3.session.Session().region_name\n", "\n", "os.environ[\"BUCKET\"] = data_bucket\n", "os.environ[\"REGION\"] = region\n", "role = sm.get_execution_role()\n", "\n", "print(f\"SageMaker role is: {role}\\nDefault SageMaker Bucket: s3://{data_bucket}\")\n", "\n", "s3=boto3.client('s3')\n", "rekognition=boto3.client('rekognition', region_name=region)\n", "sagemaker = boto3.client('sagemaker')\n", "a2i_runtime_client = boto3.client('sagemaker-a2i-runtime', region_name=region)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this lab, we will use a sample image stored in this repo: ../datasets/yoga_swimwear.jpg to test Rekognition Image moderation API.\n", "\n", "The image contains a lady in a bikini and Rekognition Image Moderation will label it as \"Suggestive.\" " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![image-moderation-hitl-arc](../datasets/yoga_swimwear.jpg)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's upload the image to the default S3 bucket for Rekognition to access." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3_key = 'content-moderation-im/image-moderation/yoga_swimwear.jpg'\n", "s3.upload_file('../datasets/yoga_swimwear.jpg', data_bucket, s3_key)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Step 2: Moderate image using Rekognition image moderation API \n", "Call Rekognition Content Moderation API to detect inappropriate information in the image." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "detectModerationLabelsResponse = rekognition.detect_moderation_labels(\n", " Image={\n", " 'S3Object': {\n", " 'Bucket': data_bucket,\n", " 'Name': s3_key,\n", " }\n", " }\n", ")\n", "detectModerationLabelsResponse" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see in the Rekognition response, the Image Moderation API labeled the image in 3 categories with confidence scores:\n", "- Top level category: **Suggestive** with a confidence score > 98%\n", "- Second level category: **Female Swimwear Or Underwear** with a confidence score > 98%\n", "- Second level category: **Revealing Clothes** with a confidence score ~ 67% \n", "\n", "The first 2 labels got high confidence scores > 98%. But the 3rd one, under Revealing Clothes, which has a low confidence score, may need a human to review and make a decision." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Step 3: Set up A2I for human review " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Setup Bucket CORS\n", "\n", "**Important**: The bucket you specify for `data_bucket` must have CORS enabled. You can enable CORS by adding a policy similar to the following to your Amazon S3 bucket. To learn how to add CORS to an S3 bucket, see [CORS Permission Requirement](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-permissions-security.html#a2i-cors-update) in the Amazon A2I documentation. \n", "\n", "\n", "```\n", "[{\n", " \"AllowedHeaders\": [],\n", " \"AllowedMethods\": [\"GET\"],\n", " \"AllowedOrigins\": [\"*\"],\n", " \"ExposeHeaders\": []\n", "}]\n", "```\n", "\n", "If you do not add a CORS configuration to the S3 buckets that contains your image input data, human review tasks for those input data objects will fail. \n", "\n", "Print out the default S3 bucket name using the below cell if you don't know which bucket is in use." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('The default SageMaker Studio S3 bucket name: ' + data_bucket)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run the below cell to enable CORS to the default S3 bucket" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cors_configuration = {\n", " 'CORSRules': [{\n", " \"AllowedHeaders\": [],\n", " \"AllowedMethods\": [\"GET\"],\n", " \"AllowedOrigins\": [\"*\"],\n", " \"ExposeHeaders\": []\n", " }]\n", "}\n", "\n", "s3.put_bucket_cors(Bucket=data_bucket,\n", " CORSConfiguration=cors_configuration)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3.1: Create Work Team \n", " A workforce is the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce for human reviews. Whichever workforce type you choose, Amazon Augmented AI takes care of sending tasks to workers. When you use a private workforce, you also create work teams, a group of workers from your workforce that are assigned to Amazon Augmented AI human review tasks. You can have multiple work teams and can assign one or more work teams to each job. \n", "\n", "This lab assumes that you already have the workforce team in the same region. If you don't have a workforce team in the current region, follow [Step 1](https://catalog.us-east-1.prod.workshops.aws/workshops/1ece9ffd-4c24-4e66-b42a-0c0e13b0f668/en-US/content-moderation/01-image-moderation/02-image-moderation-with-a2i#step-1:-create-a-private-team-in-aws-console-(you-can-skip-this-step-if-you-already-have-a-private-work-team-in-the-region)) and [Step 2](https://catalog.us-east-1.prod.workshops.aws/workshops/1ece9ffd-4c24-4e66-b42a-0c0e13b0f668/en-US/content-moderation/01-image-moderation/02-image-moderation-with-a2i#step-2:-activate-a2i-user-account) on the Workshop website to set it up in the AWS console.\n", "\n", "[More documents](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html ) about how to set up a workforce team for A2I." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# get the existing workforce arn\n", "work_team_arn = sagemaker.list_workteams()[\"Workteams\"][0][\"WorkteamArn\"]\n", "work_team_arn" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3.2: Create Human Task UI \n", "Create a human task UI resource, giving a UI template in liquid html. This template will be rendered to the human workers whenever human loop is required. We are providing a simple demo template that is compatible with AWS Rekogition's Image Moderation API input and response. Since we are integrating A2I with Rekognition, we can create the template in the Console using default templates provided by [A2I](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-instructions-overview.html). To make things easier, the below template string is copied from the defeault template provided by Amazon A2I (found in the SageMaker Console under Worker task templates)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "template = \"\"\"\n", "\n", "{% capture s3_uri %}s3://{{ task.input.aiServiceRequest.image.s3Object.bucket }}/{{ task.input.aiServiceRequest.image.s3Object.name }}{% endcapture %}\n", "\n", "\n", " \n", "

Review the image and choose all applicable categories.

If no categories apply, choose None.


Nudity

Visuals depicting nude male or female person or persons


Graphic Male Nudity

Visuals depicting full frontal male nudity, often close ups


Graphic Female Nudity

Visuals depicting full frontal female nudity, often close ups


Sexual Activity

Visuals depicting various types of explicit sexual activities and pornography


Illustrated Explicit Nudity

Visuals depicting animated or drawn sexual activity, nudity or pornography


Adult Toys

Visuals depicting adult toys, often in a marketing context


Female Swimwear or Underwear

Visuals depicting female person wearing only swimwear or underwear


Male Swimwear Or Underwear

Visuals depicting male person wearing only swimwear or underwear


Barechested Male

Visuals depicting topless males


Partial Nudity

Visuals depicting covered up nudity, for example using hands or pose


Sexual Situations

Visuals depicting passionate kissing and embracing of a sexual nature


Revealing Clothes

Visuals depicting revealing clothes and poses, such as deep cut dresses


Graphic Violence or Gore

Visuals depicting prominent blood or bloody injuries


Physical Violence

Visuals depicting violent physical assault, such as kicking or punching


Weapon Violence

Visuals depicting violence using weapons like firearms or blades, such as shooting


Weapons

Visuals depicting weapons like firearms and blades


Self Injury

Visuals depicting self-inflicted cutting on the body, typically in distinctive patterns using sharp objects


Emaciated Bodies

Visuals depicting extremely malnourished human bodies


Corpses

Visuals depicting human dead bodies


Hanging

Visuals depicting death by hanging

Air Crash Visuals depicting air crashes Explosions and Blasts Visuals depicting blasts and explosions Middle Finger Visuals depicting a person showing the middle finger as a rude gesture Drug Products Visuals depicting drug products like joints or marijuana Drug Use Visuals depicting drug use, for example, snorting drug powders Pills Visuals depicting pills of any kind Drug Paraphernalia Visuals depicting drug paraphernalia like bongs and vaporizers Tobacco Products Visuals depicting tobacco products like cigarettes and e-cigarette devices Smoking Visuals depicting a person or persons smoking Drinking Visuals depicting a person or persons drinking alcoholic beverages Alcoholic Beverages Visuals depicting bottles or containers of alcoholic beverages Gambling Visuals depicting gambling, such as slot machines or casinos Nazi Party Visuals depicting Nazi party symbols, such as the Nazi Swastika White Supremacy Visuals depicting white supremacy symbols, such as the Confederate flag Extremist Visuals depicting flags and emblems of extremist organizations

\n", "\n", " \n", " \n", "\n", "\n", "\"\"\"\n", "\n", "def create_task_ui(task_ui_name):\n", " '''\n", " Creates a Human Task UI resource.\n", "\n", " Returns:\n", " struct: HumanTaskUiArn\n", " '''\n", " response = sagemaker.create_human_task_ui(\n", " HumanTaskUiName=task_ui_name,\n", " UiTemplate={'Content': template})\n", " return response" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n", "taskUIName = 'content-moderation-im-image-ui-template'\n", "\n", "# Create task UI\n", "humanTaskUiResponse = create_task_ui(taskUIName)\n", "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n", "print(humanTaskUiArn)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3.3: Creating the Flow Definition \n", "In this section, we're going to create a flow definition. Flow Definitions allow us to specify:\n", "- The conditions under which your human loop will be called. \n", "- The workforce that your tasks will be sent to. \n", "- The instructions that your workforce will receive. This is called a worker task template. \n", "- The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks. Where your output data will be stored.\n", "\n", "This demo is going to use the API, but you can optionally create this workflow definition in the console as well. For more details and instructions, see [this document](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Specify Human Loop Activation Conditions\n", "Since we are using a built-in integration type for A2I, we can use Human Loop Activation Conditions to provide conditions that trigger a human loop.Here we are specifying conditions for specific labels in our image. If Rekognition's confidence falls outside of the thresholds set here, the imagee will be sent to a human for review, with the specific keys that triggered the human loop prompted to the worker. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def create_flow_definition(flow_definition_name, a2i_output_path):\n", " '''\n", " Creates a Flow Definition resource\n", "\n", " Returns:\n", " struct: FlowDefinitionArn\n", " '''\n", " # Visit https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-human-fallback-conditions-json-schema.html for more information on this schema.\n", " humanLoopActivationConditions = json.dumps(\n", " {\n", " \"Conditions\": [\n", " {\n", " \"And\": [\n", " {\n", " \"ConditionType\": \"ModerationLabelConfidenceCheck\",\n", " \"ConditionParameters\": {\n", " \"ModerationLabelName\": \"*\",\n", " \"ConfidenceLessThan\": 100\n", " }\n", " },\n", " {\n", " \"ConditionType\": \"ModerationLabelConfidenceCheck\",\n", " \"ConditionParameters\": {\n", " \"ModerationLabelName\": \"*\",\n", " \"ConfidenceGreaterThan\": 50\n", " }\n", " }\n", " ]\n", " }\n", " ]\n", " }\n", " )\n", "\n", " response = sagemaker.create_flow_definition(\n", " FlowDefinitionName= flow_definition_name,\n", " RoleArn= role,\n", " HumanLoopConfig= {\n", " \"WorkteamArn\": work_team_arn,\n", " \"HumanTaskUiArn\": humanTaskUiArn,\n", " \"TaskCount\": 1,\n", " \"TaskDescription\": \"Image moderation sample task description\",\n", " \"TaskTitle\": \"Image moderation sample task\"\n", " },\n", " HumanLoopRequestSource={\n", " \"AwsManagedHumanLoopRequestSource\": \"AWS/Rekognition/DetectModerationLabels/Image/V3\"\n", " },\n", " HumanLoopActivationConfig={\n", " \"HumanLoopActivationConditionsConfig\": {\n", " \"HumanLoopActivationConditions\": humanLoopActivationConditions\n", " }\n", " },\n", " OutputConfig={\n", " \"S3OutputPath\" : a2i_output_path\n", " }\n", " )\n", " \n", " return response['FlowDefinitionArn']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we are ready to create our Flow Definition!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n", "uniqueId = str(uuid.uuid4())\n", "flowDefinitionName = f'fd-rekognition-image-{uniqueId}' \n", "a2i_output_path = f's3://{data_bucket}/content-moderation-im/a2i-output' # Where A2I keeps the output data\n", "\n", "flowDefinitionArn = create_flow_definition(flowDefinitionName, a2i_output_path)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def describe_flow_definition(name):\n", " '''\n", " Describes Flow Definition\n", "\n", " Returns:\n", " struct: response from DescribeFlowDefinition API invocation\n", " '''\n", " return sagemaker.describe_flow_definition(\n", " FlowDefinitionName=name)\n", "\n", "import time\n", "# Describe flow definition - status should be active\n", "for x in range(60):\n", " describeFlowDefinitionResponse = describe_flow_definition(flowDefinitionName)\n", " print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n", " if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n", " print(\"Flow Definition is active\")\n", " break\n", " time.sleep(2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Step 4: Moderate image with human review \n", "\n", "Let's call the Rekognition Image Moderation API with Human Loop Configurations. The condition we set up earlier was to trigger A2I when any label has a confidence score greater than 50%. So it is guaranteed that we can review the image in the A2I console." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "detectModerationLabelsResponse = rekognition.detect_moderation_labels(\n", " Image={\n", " 'S3Object': {\n", " 'Bucket': data_bucket,\n", " 'Name': s3_key,\n", " }\n", " },\n", " HumanLoopConfig={\n", " \"FlowDefinitionArn\":flowDefinitionArn,\n", " \"HumanLoopName\": f\"rek-default-loop-{str(uuid.uuid4())}\",\n", " \"DataAttributes\":{\"ContentClassifiers\":[\"FreeOfPersonallyIdentifiableInformation\"]}\n", " }\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Check Status of Human Loop" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "arr = detectModerationLabelsResponse[\"HumanLoopActivationOutput\"][\"HumanLoopArn\"].split('/')\n", "human_loop_name=arr[len(arr)-1]\n", "\n", "describeHumanLoopResponse = a2i_runtime_client.describe_human_loop(HumanLoopName=human_loop_name)\n", "print(f'\\nHuman Loop Name: {human_loop_name}')\n", "print(f'Human Loop Status: {describeHumanLoopResponse[\"HumanLoopStatus\"]}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A human loop task is now sent to A2I and ready for us to review. You can find the A2I console URL from SageMaker console under Augment AI -> Human review workforces under the \"private\" tab. \n", "\n", "Or use the below code to print out the A2I console URL:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "workteamName = work_team_arn[work_team_arn.rfind('/') + 1:]\n", "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n", "print('https://' + sagemaker.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Login to the A2I console using the username/password you received earlier when setting up the Workforce team. You should see the task \"Image moderation sample task\" in the list. Choose the item and click on the \"start working\" button on the top right to start viewing the moderation result. You will see a review page like the below screenshot:\n", "\n", "![a2i image moderation](../images/image-moderation-hitl-a2i.png)\n", "\n", "As a reviewer, you can confirm the moderation categories, then click on the \"Submit\" button. A2I will store the reviewer input in the S3 output folder." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "a2i_resp = a2i_runtime_client.describe_human_loop(HumanLoopName=human_loop_name)\n", "print(\"Human Loop task status: \", a2i_resp[\"HumanLoopStatus\"])\n", "print(\"Human Loop output: \", a2i_resp[\"HumanLoopOutput\"][\"OutputS3Uri\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Step 5 Check A2I generated JSON \n", "Now, let's download the A2I output file and print it out:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3.download_file(data_bucket, a2i_resp[\"HumanLoopOutput\"][\"OutputS3Uri\"].replace(f's3://{data_bucket}/',''), 'a2i-output-image-moderation.json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `humanAnswers` field in the JSON file contains the reviewer's input. The `inputContent` field contains the original data sent to A2I." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "with open('a2i-output-image-moderation.json','r') as f:\n", " print(json.dumps(json.loads(f.read()), indent=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Conclusion\n", "In this lab, we moderated an image using Rekognition Image Moderation API. And set up the A2I components to review the moderation result." ] } ], "metadata": { "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" }, "vscode": { "interpreter": { "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49" } } }, "nbformat": 4, "nbformat_minor": 4 }