{
"cells": [
{
"cell_type": "markdown",
"id": "20d791a0",
"metadata": {
"tags": []
},
"source": [
"# Car Tracker \n",
"\n",
"\n",
"This notebook shows how to build a simple car dwell time tracker a pretrained SSD Mobilenet Tensorflow Object Detection Model and a Centroid Tracker.\n",
"\n",
"By completing this notebook, you will learn:\n",
"* How to write a Python script for your app that takes in camera streams, performs inference, and outputs results\n",
"* How to use a Tensorflow Object Detecrtion model with Panorama\n",
"* How to bundle additional Python files and libraries with your container\n",
"* How to build a simple car tracker\n",
"* How to test your code using the Test Utility, which saves you build and deploy time\n",
"* How to programmatically package and deploy applications using the Panorama CLI\n",
"\n",
"--- \n",
"\n",
"1. [Prerequisites](#Prerequisites)\n",
"1. [Set up](#Set-up)\n",
"1. [Import model](#Import-model)\n",
"1. [Write and test app code](#Write-and-test-app-code)\n",
"1. [Package app](#Package-app)\n",
"1. [Deploy app to device](#Deploy-app-to-device)"
]
},
{
"cell_type": "markdown",
"id": "6d28ad3d",
"metadata": {},
"source": [
"# Prerequisites\n",
"\n",
"1. In a terminal session on this Jupyter notebook server, run `aws configure`. This allows this notebook server to access Panorama resources on your behalf."
]
},
{
"cell_type": "markdown",
"id": "978aba4d",
"metadata": {},
"source": [
"# Set Up\n",
"Import libraries for use with this notebook environment, you do not need these libraries when you write your application code. Run these 3 cells every time you update your app code and restart your kernel."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b3214491",
"metadata": {},
"outputs": [],
"source": [
"import sys\n",
"import os\n",
"import time\n",
"import json\n",
"\n",
"import boto3\n",
"import sagemaker\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from IPython.core.magic import register_cell_magic\n",
"\n",
"sys.path.insert( 0, os.path.abspath( \"../common/test_utility\" ) )\n",
"import panorama_test_utility\n",
"\n",
"# instantiate boto3 clients\n",
"s3_client = boto3.client('s3')\n",
"panorama_client = boto3.client('panorama')\n",
"\n",
"# configure matplotlib\n",
"%matplotlib inline\n",
"plt.rcParams[\"figure.figsize\"] = (20,20)\n",
"\n",
"# register custom magic command\n",
"@register_cell_magic\n",
"def save_cell(line, cell):\n",
" 'Save python code block to a file'\n",
" with open(line, 'wt') as fd:\n",
" fd.write(cell)"
]
},
{
"cell_type": "markdown",
"id": "15beab6c",
"metadata": {
"tags": []
},
"source": [
"## Notebook parameters\n",
"\n",
"Global constants that help the notebook create Panorama resources on your behalf."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5b6fac93",
"metadata": {},
"outputs": [],
"source": [
"# Device ID, should look like: device-oc66nax4cgzwhyuaeyifrqowue\n",
"DEVICE_ID = input( 'DEVICE_ID (format: device-*)' ).strip()\n",
"\n",
"# Enter your S3 bucket info here\n",
"S3_BUCKET = input( 'S3_BUCKET' ).strip()\n",
"\n",
"# Enter your desired AWS region\n",
"AWS_REGION = input( 'AWS_REGION (e.g. us-east-1)' ).strip() \n",
"\n",
"ML_MODEL_FNAME = 'ssd_mobilenet_v2_coco'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "da90acb0",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# application name\n",
"app_name = 'car_tracker_app'\n",
"\n",
"## package names and node names\n",
"code_package_name = 'CAR_TRACKER_CODE'\n",
"model_package_name = 'CAR_SSD_TF_MODEL'\n",
"camera_node_name = 'RTSP_STREAM'\n",
"\n",
"# model node name, raw model path (without platform dependent suffics), and input data shape\n",
"model_node_name = \"model_node\"\n",
"model_file_basename = \"./models/\" + ML_MODEL_FNAME\n",
"model_data_shape = '{\"image_tensor\":[1,300,300,3]}'\n",
"\n",
"# video filename to simulate camera stream\n",
"videoname = '../common/test_utility/videos/Car-in-the-mist_coverr.mp4'\n",
"\n",
"# AWS account ID\n",
"account_id = boto3.client(\"sts\").get_caller_identity()[\"Account\"]\n"
]
},
{
"cell_type": "markdown",
"id": "d10538b4",
"metadata": {},
"source": [
"## Set up application\n",
"\n",
"Every application uses the creator's AWS Account ID as the prefix to uniquely identifies the application resources. Running `panorama-cli import-application` replaces the generic account Id with your account Id."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d017f0bd",
"metadata": {},
"outputs": [],
"source": [
"!cd ./car_tracker_app && panorama-cli import-application"
]
},
{
"cell_type": "markdown",
"id": "8933b6d8",
"metadata": {
"tags": []
},
"source": [
"# Import model\n",
"\n",
"We need to compile and import the model twice. Once for testing with this notebook server and once for deploying to the Panorama device.\n",
"\n",
"While working with the Panorama sample code, we provide pretrained models for you to use. Locally, models are stored in `./models`. This step downloads the model artifacts from our Amazon S3 bucket to the local folder. If you want to use your own models, put your tar.gz file into the `./models` folder."
]
},
{
"cell_type": "markdown",
"id": "23f9a327",
"metadata": {},
"source": [
"### Prepare model for testing with notebook server"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ca75fc18",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"# Downloads pretrained model for this sample.\n",
"# This step takes some time, depending on your network environment.\n",
"panorama_test_utility.download_sample_model( ML_MODEL_FNAME, \"./models\" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ac71b397",
"metadata": {},
"outputs": [],
"source": [
"# Compile the model to run with test-utility.\n",
"# This step takes 30 mins ~ 40 mins.\n",
"%run ../common/test_utility/panorama_test_utility_compile.py \\\n",
"\\\n",
"--s3-model-location s3://{S3_BUCKET}/{app_name}/ \\\n",
"\\\n",
"--model-node-name model_node \\\n",
"--model-file-basename ./models/{ML_MODEL_FNAME} \\\n",
"--model-data-shape '{model_data_shape}' \\\n",
"--model-framework TENSORFLOW"
]
},
{
"cell_type": "markdown",
"id": "e17a5f4a",
"metadata": {
"tags": []
},
"source": [
"### Prepare model for deploying to Panorama device\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b01d42e3",
"metadata": {},
"outputs": [],
"source": [
"model_asset_name = 'model_asset'\n",
"model_package_path = f'packages/{account_id}-{model_package_name}-1.0'\n",
"model_descriptor_path = f'packages/{account_id}-{model_package_name}-1.0/descriptor.json'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d347cc0f",
"metadata": {},
"outputs": [],
"source": [
"!cd ./car_tracker_app && panorama-cli add-raw-model \\\n",
" --model-asset-name {model_asset_name} \\\n",
" --model-s3-uri s3://{S3_BUCKET}/{app_name}/{model_node_name}/{ML_MODEL_FNAME}.tar.gz \\\n",
" --descriptor-path {model_descriptor_path} \\\n",
" --packages-path {model_package_path}"
]
},
{
"cell_type": "markdown",
"id": "db9ba77e",
"metadata": {},
"source": [
"# Write and test app code"
]
},
{
"cell_type": "markdown",
"id": "e11fbf47",
"metadata": {},
"source": [
"Every app has an entry point script, written in Python that pulls the frames from camera streams, performs inference, and send the results to the desired location. This file can be found in `your_app/packages/code_node/src/app.py`. Below, you will iterate on the code from within the notebook environment. The entry point file will be updated everytime you run the next notebook cell thanks to the `%%save_cell`. This is a magic command to update the contents of the entry point script. \n",
"\n",
"After updating the entry point script, use the Test Utility Run command (panorama_test_utility_run.py) command to simulate the application.\n",
"\n",
"### Iterating on Code Changes\n",
"\n",
"To iterate on the code:\n",
"1. Interrupt the kernel if application is still running.\n",
"2. Make changes in the next cell, and run the cell to update the entry point script. \n",
"3. Run the panorama_test_utility_run.py again.\n",
"\n",
"**CHANGE VIDEO** : For you to change video, please set the file path to the --video-file argument of the panorama_test_utility_run.py command."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "95db2d38",
"metadata": {},
"outputs": [],
"source": [
"%%save_cell ./{app_name}/packages/{account_id}-{code_package_name}-1.0/src/app.py\n",
"\n",
"import json\n",
"import logging\n",
"import time\n",
"from logging.handlers import RotatingFileHandler\n",
"\n",
"import boto3\n",
"import cv2\n",
"import numpy as np\n",
"import panoramasdk\n",
"import datetime\n",
"from CentroidTracker import CentroidTracker\n",
"\n",
"class Application(panoramasdk.node):\n",
" def __init__(self):\n",
" \"\"\"Initializes the application's attributes with parameters from the interface, and default values.\"\"\"\n",
"\n",
" self.MODEL_NODE = \"model_node\"\n",
" self.MODEL_DIM = 300\n",
" self.frame_num = 0\n",
" self.tracker = CentroidTracker(maxDisappeared=80, maxDistance=90)\n",
" self.tracked_objects = []\n",
" self.tracked_objects_start_time = dict()\n",
" self.tracked_objects_duration = dict()\n",
"\n",
" try:\n",
" # Parameters\n",
" logger.info('Configuring parameters.')\n",
" self.threshold = self.inputs.threshold.get()\n",
" \n",
" # Desired class\n",
" self.classids = [3.]\n",
"\n",
" except:\n",
" logger.exception('Error during initialization.')\n",
" finally:\n",
" logger.info('Initialiation complete.')\n",
"\n",
" def process_streams(self):\n",
" \"\"\"Processes one frame of video from one or more video streams.\"\"\"\n",
" self.frame_num += 1\n",
" logger.debug(self.frame_num)\n",
"\n",
" # Loop through attached video streams\n",
" streams = self.inputs.video_in.get()\n",
" for stream in streams:\n",
" self.process_media(stream)\n",
"\n",
" self.outputs.video_out.put(streams)\n",
"\n",
" def process_media(self, stream):\n",
" \"\"\"Runs inference on a frame of video.\"\"\"\n",
" image_data = preprocess(stream.image, self.MODEL_DIM)\n",
" logger.debug(image_data.shape)\n",
"\n",
" # Run inference\n",
" inference_results = self.call({\"image_tensor\":image_data}, self.MODEL_NODE)\n",
"\n",
" # Process results (object deteciton)\n",
" self.process_results(inference_results, stream)\n",
"\n",
" def process_results(self, inference_results, stream):\n",
" \"\"\"Processes output tensors from a computer vision model and annotates a video frame.\"\"\"\n",
" if inference_results is None:\n",
" logger.warning(\"Inference results are None.\")\n",
" return\n",
" \n",
" w,h,c = stream.image.shape\n",
"\n",
" conf_scores = None\n",
" classes = None\n",
" bboxes = None\n",
" rects = []\n",
"\n",
" for det in inference_results:\n",
" if det.shape[-1] == 4:\n",
" bboxes = det[0]\n",
" elif det.shape[-1] == 100:\n",
" if det[0][0] >= 1:\n",
" classes = det[0]\n",
" else:\n",
" conf_scores = det[0]\n",
" \n",
" for a in range(len(conf_scores)):\n",
" if conf_scores[a] * 100 > self.threshold and classes[a] in self.classids:\n",
" (top, left, bottom, right) = bboxes[a]\n",
" rects.append([left*w, top*h, right*w, bottom*h])\n",
" stream.add_rect(left, top, right, bottom)\n",
" \n",
" rects = np.array(rects)\n",
" rects = rects.astype(int)\n",
" objects = self.tracker.update(rects)\n",
" \n",
" logger.info('Tracking {} cars'.format(len(objects)))\n",
" \n",
" for (objectID, bbox) in objects.items():\n",
" x1, y1, x2, y2 = bbox\n",
" x1 = int(x1)\n",
" y1 = int(y1)\n",
" x2 = int(x2)\n",
" y2 = int(y2)\n",
"\n",
" if objectID not in self.tracked_objects:\n",
" self.tracked_objects.append(objectID)\n",
" self.tracked_objects_start_time[objectID] = datetime.datetime.now()\n",
" self.tracked_objects_duration[objectID] = 0\n",
" else:\n",
" time_diff = datetime.datetime.now() - self.tracked_objects_start_time[objectID]\n",
" sec = time_diff.total_seconds()\n",
" self.tracked_objects_duration[objectID] = sec\n",
" \n",
" duration = self.tracked_objects_duration[objectID]\n",
" \n",
" logger.info('CarId: {} at ({},{}) for {}'.format(objectID, x1, y1, duration))\n",
" stream.add_rect(x1/w, y1/h, x2/w, y2/h)\n",
" stream.add_label('{}s'.format(str(duration)), x1/w, y1/h)\n",
"\n",
"def preprocess(img, size):\n",
" \"\"\"Resizes and normalizes a frame of video.\"\"\"\n",
" resized = cv2.resize(img, (size, size))\n",
" x1 = np.asarray(resized)\n",
" x1 = np.expand_dims(x1, 0)\n",
" return x1\n",
"\n",
"def get_logger(name=__name__,level=logging.INFO):\n",
" logger = logging.getLogger(name)\n",
" logger.setLevel(level)\n",
" handler = RotatingFileHandler(\"/opt/aws/panorama/logs/app.log\", maxBytes=100000000, backupCount=2)\n",
" formatter = logging.Formatter(fmt='%(asctime)s %(levelname)-8s %(message)s',\n",
" datefmt='%Y-%m-%d %H:%M:%S')\n",
" handler.setFormatter(formatter)\n",
" logger.addHandler(handler)\n",
" return logger\n",
"\n",
"def main():\n",
" try:\n",
" logger.info(\"INITIALIZING APPLICATION\")\n",
" app = Application()\n",
" logger.info(\"PROCESSING STREAMS\")\n",
" while True:\n",
" app.process_streams()\n",
" except:\n",
" logger.exception('Exception during processing loop.')\n",
"\n",
"logger = get_logger(level=logging.INFO)\n",
"main()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cac28e64-3f58-4bc8-a582-2dac6895ac9b",
"metadata": {},
"outputs": [],
"source": [
"# Run the application with test-utility.\n",
"#\n",
"# As '--output-pyplot' option is specified, this command simulates HDMI output with pyplot rendering in the output cell.\n",
"# In order to see console output (stdout/stderr) from the application, please remove the --output-pyplot option.\n",
"#\n",
"%run ../common/test_utility/panorama_test_utility_run.py \\\n",
"\\\n",
"--app-name {app_name} \\\n",
"--code-package-name {code_package_name} \\\n",
"--model-package-name {model_package_name} \\\n",
"--camera-node-name {camera_node_name} \\\n",
"--model-node-name {model_node_name} \\\n",
"--model-file-basename {model_file_basename} \\\n",
"--video-file {videoname} \\\n",
"--py-file ./{app_name}/packages/{account_id}-{code_package_name}-1.0/src/app.py \\\n",
"--output-pyplot"
]
},
{
"cell_type": "markdown",
"id": "b9901ae3",
"metadata": {},
"source": [
"# Package app\n",
"\n",
"Updates the app to be deployed with the recent code"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f72c161f",
"metadata": {},
"outputs": [],
"source": [
"py_file_name = 'app.py'\n",
"panorama_test_utility.update_package_descriptor( app_name, account_id, code_package_name, py_file_name )"
]
},
{
"cell_type": "markdown",
"id": "bb318878",
"metadata": {},
"source": [
"### Build app with container"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "80b6c66c",
"metadata": {},
"outputs": [],
"source": [
"container_asset_name = 'code_asset'"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "783e965b",
"metadata": {},
"outputs": [],
"source": [
"%%capture captured_output\n",
"\n",
"# Building container image.This process takes time (5min ~ 10min)\n",
"# FIXME : without %%capture, browser tab crashes because of too much output from the command.\n",
"\n",
"!cd ./car_tracker_app && panorama-cli build \\\n",
" --container-asset-name {container_asset_name} \\\n",
" --package-path packages/{account_id}-{code_package_name}-1.0"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4de3b376-2ad5-4fca-9512-8dddd59ff2b9",
"metadata": {},
"outputs": [],
"source": [
"stdout_lines = captured_output.stdout.splitlines()\n",
"stderr_lines = captured_output.stderr.splitlines()\n",
"print(\" :\")\n",
"print(\" :\")\n",
"for line in stdout_lines[-30:] + stderr_lines[-30:]:\n",
" print(line)"
]
},
{
"cell_type": "markdown",
"id": "e1a21ace",
"metadata": {},
"source": [
"### Upload application to Panorama for deploying to devices"
]
},
{
"cell_type": "markdown",
"id": "a5a6207c-54af-4f98-a001-60ee247a0c7c",
"metadata": {},
"source": [
"### Update camera streams\n",
"\n",
"In the AWS Panorama console, you can select the camera streams, but programmatically, you need to define the camera stream info for the cameras you are using with the app.\n",
"\n",
"Open the ```package.json``` in ```packages/<account_number>-RTSP_STREAM-1.0``` and update the camera username, password and URL. After you have updated your camera credentials, run package-application. You can override this camera stream when you deploy the app."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "70ed5aa3",
"metadata": {},
"outputs": [],
"source": [
"# This step takes some time, depending on your network environment.\n",
"!cd ./car_tracker_app && panorama-cli package-application"
]
},
{
"cell_type": "markdown",
"id": "55b10936",
"metadata": {},
"source": [
"### Ready for deploying to a device\n",
"\n",
"Congrats! Your app is now ready to deploy to a device. Next, you can continue in this notebook to deploy the app programmatically or you can go to the Panorama console and deploying using a GUI. The GUI makes it easier to select camera streams and select the devices you want to deploy to. Programmatic deployment is faster to complete and easier to automate.\n",
"\n",
"# Deploy app to device\n",
"\n",
"Let's make sure the device we are deploying to is available."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "38cc2770",
"metadata": {},
"outputs": [],
"source": [
"response = panorama_client.describe_device(\n",
" DeviceId= DEVICE_ID\n",
")\n",
"\n",
"print('You are deploying to Device: {}'.format(response['Name']))"
]
},
{
"cell_type": "markdown",
"id": "9d82cc29-5089-4110-a73c-950e9c4d4c9c",
"metadata": {},
"source": [
"#### Deploy Application"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "922c3227",
"metadata": {},
"outputs": [],
"source": [
"response = panorama_test_utility.deploy_app(DEVICE_ID, app_name, role=None)\n",
"\n",
"application_instance_id = response['ApplicationInstanceId']\n",
"\n",
"response"
]
},
{
"cell_type": "markdown",
"id": "51ed70c2-91af-4402-b179-2673c884a3e2",
"metadata": {},
"source": [
"# Clean up"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31b5fd3a",
"metadata": {},
"outputs": [],
"source": [
"panorama_test_utility.remove_application( DEVICE_ID, application_instance_id )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "55d6c9f9-42c5-4386-97f1-37adf4a627ff",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5"
}
},
"nbformat": 4,
"nbformat_minor": 5
}