{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "a73bd45f-9f55-4c7b-93ae-9db1135f2f0f", "metadata": { "tags": [] }, "source": [ "# Amanzon SageMaker Ground Truth Demonstration for Video Frame Object Tracking Labeling Job\n", "\n", "1. [Introduction](#1-introduction)\n", " 1. [Cost and Runtime](#11-cost-and-runtime)\n", " 2. [Prerequisites](#12-prerequisites)\n", "2. [Launch the Notebook Instance and Setup the Environment](#2-launch-the-notebook-instance-and-setup-the-environment)\n", "3. [Run a Ground Truth Labeling Job](#3-run-a-ground-truth-labeling-job)\n", " 1. [Prepare the Data](#31-prepare-the-data)\n", " 2. [Create a Video Frame Input Manifest File](#32-create-a-video-frame-input-manifest-file)\n", " 3. [Create an Instruction Template](#33-create-the-instruction-template)\n", " 4. [Use a private team to test your task](#Use-a-private-team-to-test-your-task)\n", " 5. [Define Pre-built Lambda Functions for Use In the Labeling Job](#35-define-pre-built-lambda-functions-for-use-in-the-labeling-job)\n", " 6. [Submit the Ground Truth job request](#36-submit-the-ground-truth-job-request)\n", " 7. [Monitor the Job Progress](#37-monitor-the-job-progress)\n", " 8. [Preview the Worker UI Task](#38-preview-the-worker-ui-task)\n", " 9. [View the Task Results](#39-view-the-task-results)\n", "4. [Clean Up - Optional](#4-clean-up---optional)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "0abbbc7c-a515-4475-934f-c48cf2c66b48", "metadata": {}, "source": [ "## 1. Introduction\n", "\n", "This sample notebook takes you through an end-to-end workflow to demonstrate the functionality of SageMaker Ground Truth Video Frame Object Tracking. You can use the video frame object tracking task type to have workers track the movement of objects in a sequence of video frames (images extracted from a video) using bounding boxes, polylines, polygons or keypoint annotation tools.\n", "\n", "Before you begin, we highly recommend you start a Ground Truth labeling job through the AWS Console first to familiarize yourself with the workflow. The AWS Console offers less flexibility than the API, but is simple to use.\n", "\n", "For more information, refer to Amazon SageMaker Developer Guide: [Video Frame Object Tracking](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-video-object-tracking.html).\n", "\n", "#### 1.1 Cost and Runtime\n", "\n", "1. For pricing, please refer to [Ground Truth pricing scheme](https://aws.amazon.com/sagemaker/groundtruth/pricing/). In order to reduce the cost, we will use Ground Truth's auto-labeling feature. Amazon SageMaker Ground Truth can use active learning to automate the labeling of your input data for certain built-in task types. Active learning is a machine learning technique that identifies data that should be labeled by your workers. In Ground Truth, this functionality is called automated data labeling. Automated data labeling helps to reduce the cost and time that it takes to label your dataset compared to using only humans.\n", "\n", "#### 1.2 Prerequisites\n", "To run this notebook, you can simply execute each cell one-by-one. To understand what's happening, you'll need:\n", "* An S3 bucket you can write to -- please provide its name in the following cell. The bucket must be in the same region as this SageMaker Notebook instance. You can also change the `EXP_NAME` to any valid S3 prefix. All the files related to this experiment will be stored in that prefix of your bucket.\n", "* Basic familiarity with [AWS S3](https://docs.aws.amazon.com/s3/index.html)\n", "* Basic understanding of [AWS Sagemaker](https://aws.amazon.com/sagemaker/)\n", "* Basic familiarity with [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/). Set it up with credentials to access the AWS account you're running this notebook from. This should work out-of-the-box on SageMaker Jupyter Notebook instances." ] }, { "attachments": {}, "cell_type": "markdown", "id": "b560a438-db66-40f0-8381-98ad893d5337", "metadata": {}, "source": [ "## 2. Launch the Notebook Instance and Setup the Environment\n", "In this step, you will use Amazon SageMaker Studio notebook to call Amazon SageMaker APIs to create a video frame object tracking labeling job. In SageMaker Studio, click on \"File Browser\" pane on the left side, navigate to \"amazon-sagemaker-groundtruth-workshop/02-module-label-videos-videoframes\" directory and then double click on `video-frame-object-tracking-labeling.ipynb` notebook.\n", "If you are prompted to choose a Kernel, choose the “Python 3 (Data Science)” kernel and click “Select”.\n", "\n", "This notebook is only tested on a SageMaker Studio Notebook & SageMaker Notebook Instances. The runtimes given are approximate, we used an `ml.t3.medium` instance with `Data Science` image. However, you can also run it on a local instance by first executing the cell below on SageMaker, and then copying the `role` string to your local copy of the notebook.\n", "\n", "NOTES: \n", "- This notebook will create/remove subdirectories in its working directory. We recommend to place this notebook in its own directory before running it. \n", "\n", "- Ground Truth requires all S3 buckets that contain labeling job input image data have a CORS policy attached. To learn more about this change, see [CORS Permission Requirement](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-video-overview.html) for Video Frame Object Tracking." ] }, { "cell_type": "code", "execution_count": null, "id": "7973cd90-c64a-4df1-88e8-b82be01a2edc", "metadata": { "tags": [] }, "outputs": [], "source": [ "# cell 01\n", "\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "import os\n", "import json\n", "import time\n", "import pandas as pd\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "from sklearn.metrics import confusion_matrix\n", "import boto3\n", "import sagemaker\n", "from urllib.parse import urlparse\n", "import warnings\n", "\n", "sess = sagemaker.Session()\n", "BUCKET = sess.default_bucket() \n", "\n", "EXP_NAME = \"label-video/video-frame-object-tracking\" # Any valid S3 prefix." ] }, { "cell_type": "code", "execution_count": null, "id": "7210a79d-ca05-419d-b2a3-09cb12c62b03", "metadata": { "tags": [] }, "outputs": [], "source": [ "# cell 02\n", "\n", "# Make sure the bucket is in the same region as this notebook.\n", "role = sagemaker.get_execution_role()\n", "region = boto3.session.Session().region_name\n", "\n", "s3 = boto3.client(\"s3\")\n", "bucket_region = s3.head_bucket(Bucket=BUCKET)[\"ResponseMetadata\"][\"HTTPHeaders\"][\n", " \"x-amz-bucket-region\"\n", "]\n", "\n", "assert (\n", " bucket_region == region\n", "), f\"You S3 bucket {BUCKET} and this notebook need to be in the same region.\"" ] }, { "attachments": {}, "cell_type": "markdown", "id": "8e627465-266f-4c6d-bd2c-a64e57391731", "metadata": { "tags": [] }, "source": [ "## 3. Run a Ground Truth Labeling Job\n", "\n", "\n", "**This section should take about 30 min to complete.**\n", "\n", "We will first run a labeling job. This involves several steps: collecting the video frames for labeling, specifying the possible label categories, creating instructions, and writing a labeling job specification.\n", "\n", "### 3.1 Prepare the data\n", "\n", "For the purpose of this demo, we use 9 frames dataset that is created by the author, Michael Daniels and this dataset can be found in `object_tracking_data` directory.\n", "\n", "We will copy these frames from `object_tracking_data` directory to our local `BUCKET`, and will create the corresponding *input manifest*. The input manifest is a formatted list of the S3 locations of the images we want Ground Truth to annotate. We will upload this manifest to our S3 `BUCKET`.\n", "\n", "### 3.2 Create a Video Frame Input Manifest File\n", "Ground Truth uses the input manifest file to identify the location of your input dataset when creating labeling tasks. For video frame object object tracking labeling jobs, each line in the input manifest file identifies the location of a video frame sequence file. Each sequence file identifies the images included in a single sequence of video frames. For more information, click [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-video-manual-data-setup.html#sms-video-create-manifest). Run the next cell to create input.manifest and input.manifest.json files." ] }, { "cell_type": "code", "execution_count": null, "id": "b7a8dc88-7274-4523-a039-a92253bdf7bf", "metadata": { "tags": [] }, "outputs": [], "source": [ "# cell 03\n", "\n", "manifest_name = 'input.manifest'\n", "\n", "total_frames = 0\n", "frames = []\n", "fr_no = 0\n", "for i, filename in enumerate(sorted(os.listdir('./object_tracking_data/'))):\n", " if filename.endswith(('jpg','jpeg','png')):\n", " total_frames += 1\n", " frames.append({\"frame-no\":fr_no,\"frame\":filename})\n", " s3.upload_file(f\"./object_tracking_data/{filename}\", BUCKET, EXP_NAME + f\"/{filename}\")\n", " fr_no+=1\n", " \n", "json_body = {\n", " \"seq-no\":1,\n", " f\"prefix\":f\"s3://{BUCKET}/{EXP_NAME}/\",\n", " \"number-of-frames\":total_frames,\n", " \"frames\":frames\n", " }\n", "\n", "# Create input.manifest.json file\n", "with open(\"./input.manifest.json\", \"w\") as f:\n", " json.dump(json_body, f, separators=(',', ':'))\n", " \n", "# Create input.manifest file \n", "manifest = {\"source-ref\":f\"s3://{BUCKET}/{EXP_NAME}/{manifest_name}.json\"}\n", "\n", "with open(f\"./{manifest_name}\", \"w\") as outfile:\n", " json.dump(manifest, outfile, separators=(',', ':'))" ] }, { "attachments": {}, "cell_type": "markdown", "id": "1ffc5bde-65f5-4114-a6bb-b3ff55f15343", "metadata": {}, "source": [ "Run the next cell to upload `input.manifest` and `input.manifest.json` files to S3." ] }, { "cell_type": "code", "execution_count": null, "id": "20031da2-a4e6-4c49-8ac1-47511049dc11", "metadata": { "tags": [] }, "outputs": [], "source": [ "# cell 04\n", "\n", "s3.upload_file(\"input.manifest\", BUCKET, f\"{EXP_NAME.split('/')[0]}\" + \"/input.manifest\")\n", "s3.upload_file(\"input.manifest.json\", BUCKET, EXP_NAME + \"/input.manifest.json\")" ] }, { "attachments": {}, "cell_type": "markdown", "id": "1cfdadc9-2104-41c7-bf7f-4e68649ef998", "metadata": {}, "source": [ "After running the cell above, you should be able to see the following files in [S3 console](https://console.aws.amazon.com/s3/):\n", " \n", "- `s3://BUCKET/label-video/video-frame-object-tracking/input.manifest.json`\n", "- `s3://BUCKET/label-video/input.manifest`\n", "\n", "We recommend you inspect the contents of these content! You can download them all to a local machine using the AWS CLI." ] }, { "attachments": {}, "cell_type": "markdown", "id": "b39fd076-d195-423c-a165-61eb6fe75342", "metadata": {}, "source": [ "### Create the Instruction Template \n", " Specify labels and provide instructions for the workers" ] }, { "cell_type": "code", "execution_count": null, "id": "ae761eb9-00cf-475a-99fc-78e8855e1fb0", "metadata": { "tags": [] }, "outputs": [], "source": [ "# cell 05\n", "\n", "# define the classes\n", "json_body = {\n", " \"labels\": [\n", " {\n", " \"label\": \"cat\"\n", " }\n", " ],\n", " \"instructions\": {\n", " \"shortInstruction\": \"
Please draw bounding box for each object in each frame
\",\n", " \"fullInstruction\": \"