{ "cells": [ { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# Bird Demo\n", "\n", "\n", "This notebook shows how to build a simple bird classifier using MobileNetV2 TF2.x image classification model\n", "\n", "By completing this notebook, you will learn:\n", "* How to write a Python script for your app that takes in camera streams, performs inference, and outputs results\n", "* How to use a Tensorflow Image Classification model with Panorama\n", "* How to bundle additional Python files and libraries with your container\n", "* How to test your code using the Panorama emulator, which saves you build and deploy time\n", "* How to programmatically package and deploy applications using the Panorama CLI\n", "\n", "--- \n", "\n", "1. [Prerequisites](#Prerequisites)\n", "1. [Set up](#Set-up)\n", "1. [Import model](#Import-model)\n", "1. [Write and test app code](#Write-and-test-app-code)\n", "1. [Package app](#Package-app)\n", "1. [Deploy app to device](#Deploy-app-to-device)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Prerequisites\n", "\n", "1. In a terminal session on this Jupyter notebook server, run `aws configure`. This allows this notebook server to access Panorama resources on your behalf." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Run this only once\n", "!pip3 install scipy" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Set Up\n", "Import libraries for use with this notebook environment, you do not need these libraries when you write your application code. Run these 3 cells every time you update your app code and restart your kernel." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sys\n", "import os\n", "import time\n", "import json\n", "\n", "import boto3\n", "import sagemaker\n", "\n", "import matplotlib.pyplot as plt\n", "from IPython.core.magic import register_cell_magic\n", "\n", "sys.path.insert( 0, os.path.abspath( \"../common/test_utility\" ) )\n", "import panorama_test_utility\n", "\n", "# instantiate boto3 clients\n", "s3_client = boto3.client('s3')\n", "panorama_client = boto3.client('panorama')\n", "\n", "# configure matplotlib\n", "%matplotlib inline\n", "plt.rcParams[\"figure.figsize\"] = (20,20)\n", "\n", "# register custom magic command\n", "@register_cell_magic\n", "def save_cell(line, cell):\n", " 'Save python code block to a file'\n", " with open(line, 'wt') as fd:\n", " fd.write(cell)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Copy the test video in the correct folder directory for the test utility" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Notebook parameters\n", "\n", "Global constants that help the notebook create Panorama resources on your behalf." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Device Related Globals\n", "DEVICE_ID = 'test-device-id' # Device ID, should look like: device-oc66nax4cgzwhyuaeyifrqowue\n", "assert DEVICE_ID != '', \"PLEASE ENTER YOUR DEVICE ID\"\n", "\n", "# Bucket Related Globals\n", "S3_BUCKET = 'sagemaker-us-west-2-987720697751' # Enter your S3 bucket info here\n", "assert S3_BUCKET != '', \"PLEASE ENTER A BUCKET NAME\"\n", "\n", "# AWS region\n", "AWS_REGION = 'us-west-2' # Enter your desired AWS region\n", "\n", "# Name of your model\n", "ML_MODEL_FNAME = 'mobilenet_v2_bird'" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "# application name\n", "app_name = 'bird_demo_app'\n", "\n", "## package names and node names\n", "code_package_name = 'BIRD_DEMO_CODE'\n", "model_package_name = 'BIRD_DEMO_TF_MODEL'\n", "camera_node_name = 'RTSP_STREAM'\n", "\n", "# model node name, raw model path (without platform dependent suffics), and input data shape\n", "model_node_name = \"model_node\"\n", "model_file_basename = \"./models/\" + ML_MODEL_FNAME\n", "model_data_shape = '{\"input_1\":[1,224,224,3]}'\n", "\n", "# video filename to simulate camera stream\n", "videoname = 'bird.mp4'\n", "\n", "# AWS account ID\n", "account_id = boto3.client(\"sts\").get_caller_identity()[\"Account\"]" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Set up application\n", "\n", "Every application uses the creator's AWS Account ID as the prefix to uniquely identifies the application resources. Running `panorama-cli import-application` replaces the generic account Id with your account Id." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!cd ./bird_demo_app && panorama-cli import-application" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# Import model\n", "\n", "We need to compile and import the model twice. Once for testing with this notebook server and once for deploying to the Panorama device.\n", "\n", "While working with the Panorama sample code, we provide pretrained models for you to use. Locally, models are stored in `panorama_sdk/models`. This step downloads the model artifacts from our Amazon S3 bucket to the local folder. If you want to use your own models, put your tar.gz file into the `panorama_sdk/models folder`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Prepare model for testing with notebook server" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# put the s3 link of your model artifacts here\n", "s3_model = 's3://sagemaker-us-west-2-987720697751/8lpv6zw4nz8p-BirdEnd-bQbqFgG3A7-001-3f575f7f/output/model.tar.gz'\n", "local_path = f'./models/{ML_MODEL_FNAME}.tar.gz'\n", "!aws s3 cp $s3_model $local_path" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Compile the model to run with test-utility.\n", "# This step takes 30 mins ~ 40 mins.\n", "%run ../common/test_utility/panorama_test_utility_compile.py \\\n", "\\\n", "--s3-model-location s3://{S3_BUCKET}/{app_name}/ \\\n", "\\\n", "--model-node-name model_node \\\n", "--model-file-basename ./models/{ML_MODEL_FNAME} \\\n", "--model-data-shape '{model_data_shape}' \\\n", "--model-framework TENSORFLOW" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "### Prepare model for deploying to Panorama device\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "model_asset_name = 'model_asset'\n", "model_package_path = f'packages/{account_id}-{model_package_name}-1.0'\n", "model_descriptor_path = f'packages/{account_id}-{model_package_name}-1.0/descriptor.json'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!cd ./bird_demo_app && panorama-cli add-raw-model \\\n", " --model-asset-name {model_asset_name} \\\n", " --model-s3-uri s3://{S3_BUCKET}/{app_name}/model_node/{ML_MODEL_FNAME}.tar.gz \\\n", " --descriptor-path {model_descriptor_path} \\\n", " --packages-path {model_package_path}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Write and test app code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Every app has an entry point script, written in Python that pulls the frames from camera streams, performs inference, and send the results to the desired location. This file can be found in `your_app/packages/code_node/src/main.py`. Below, you will iterate on the code from within the notebook environment. The entry point file will be updated everytime you run the next notebook cell thanks to the `%%run_and_save app.py`. This is a utility function to update the contents of the entry point script. \n", "\n", "The next cell will loop through the test video until you select Stop on your Jupyter notebook cell. \n", "\n", "### Iterating on Code Changes\n", "\n", "To iterate on the code:\n", "1. Make changes in the next cell. \n", "2. Stop the Notebook and Reset the Kernel.\n", "3. Run the 3 Setup cells again before re-running the code cell below.\n", "\n", "**CHANGE VIDEO** : For you to change video, please place the video in samples/panorama_sdk/videos and update the global variables with the video and extension name (video.avi for example)\n", "\n", "Run only the Set up and Notebook Parameters cells before running this for reiteration" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%save_cell ./{app_name}/packages/{account_id}-{code_package_name}-1.0/src/app.py\n", "\n", "import json\n", "import logging\n", "import time\n", "from logging.handlers import RotatingFileHandler\n", "\n", "import boto3\n", "from botocore.exceptions import ClientError\n", "import cv2\n", "import numpy as np\n", "import panoramasdk\n", "import datetime\n", "\n", "class Application(panoramasdk.node):\n", " def __init__(self):\n", " \"\"\"Initializes the application's attributes with parameters from the interface, and default values.\"\"\"\n", " self.MODEL_NODE = \"model_node\"\n", " self.MODEL_DIM = 224\n", " self.frame_num = 0\n", " self.tracked_objects = []\n", " self.tracked_objects_start_time = dict()\n", " self.tracked_objects_duration = dict()\n", " \n", " self.classes = {\n", " 0: \"Bobolink\", \n", " 1: \"Cardinal\", \n", " 2: \"Purple_Finch\",\n", " 3: \"Northern_Flicker\",\n", " 4:\"American_Goldfinch\",\n", " 5:\"Ruby_throated_Hummingbird\",\n", " 6:\"Blue_Jay\",\n", " 7:\"Mallard\" \n", " }\n", "\n", " def process_streams(self):\n", " \"\"\"Processes one frame of video from one or more video streams.\"\"\"\n", " self.frame_num += 1\n", " logger.debug(self.frame_num)\n", "\n", " # Loop through attached video streams\n", " streams = self.inputs.video_in.get()\n", " for stream in streams:\n", " self.process_media(stream)\n", "\n", " self.outputs.video_out.put(streams)\n", "\n", " def process_media(self, stream):\n", " \"\"\"Runs inference on a frame of video.\"\"\"\n", " image_data = preprocess(stream.image, self.MODEL_DIM)\n", " logger.debug(image_data.shape)\n", "\n", " # Run inference\n", " inference_results = self.call({\"input_1\":image_data}, self.MODEL_NODE)\n", "\n", " # Process results (object deteciton)\n", " self.process_results(inference_results, stream)\n", "\n", " def process_results(self, inference_results, stream):\n", " \"\"\"Processes output tensors from a computer vision model and annotates a video frame.\"\"\"\n", " if inference_results is None:\n", " logger.warning(\"Inference results are None.\")\n", " return\n", " \n", " logger.debug('Inference results: {}'.format(inference_results))\n", " count = 0\n", " for det in inference_results:\n", " if count == 0:\n", " first_output = det\n", " count += 1\n", " \n", " # first_output = inference_results[0]\n", " logger.debug('Output one type: {}'.format(type(first_output)))\n", " probabilities = first_output[0]\n", " # 1000 values for 1000 classes\n", " logger.debug('Result one shape: {}'.format(probabilities.shape))\n", " top_result = probabilities.argmax()\n", " \n", " self.detected_class = self.classes[top_result]\n", " self.detected_frame = self.frame_num\n", " # persist for up to 5 seconds\n", " # if self.frame_num - self.detected_frame < 75:\n", " label = '{} ({}%)'.format(self.detected_class, int(probabilities[top_result]*100))\n", " stream.add_label(label, 0.1, 0.1)\n", "\n", "def preprocess(img, size):\n", " \"\"\"Resizes and normalizes a frame of video.\"\"\"\n", " resized = cv2.resize(img, (size, size))\n", " x1 = np.asarray(resized)\n", " x1 = np.expand_dims(x1, 0)\n", " return x1\n", "\n", "def get_logger(name=__name__,level=logging.INFO):\n", " logger = logging.getLogger(name)\n", " logger.setLevel(level)\n", " handler = RotatingFileHandler(\"/opt/aws/panorama/logs/app.log\", maxBytes=100000000, backupCount=2)\n", " formatter = logging.Formatter(fmt='%(asctime)s %(levelname)-8s %(message)s',\n", " datefmt='%Y-%m-%d %H:%M:%S')\n", " handler.setFormatter(formatter)\n", " logger.addHandler(handler)\n", " return logger\n", "\n", "def main():\n", " try:\n", " logger.info(\"INITIALIZING APPLICATION\")\n", " app = Application()\n", " logger.info(\"PROCESSING STREAMS\")\n", " while True:\n", " app.process_streams()\n", " except Exception as e:\n", " logger.warning(e)\n", "\n", "logger = get_logger(level=logging.INFO)\n", "main()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Run the application with test-utility.\n", "#\n", "# As '--output-pyplot' option is specified, this command simulates HDMI output with pyplot rendering in the output cell.\n", "# In order to see console output (stdout/stderr) from the application, please remove the --output-pyplot option.\n", "#\n", "%run ../common/test_utility/panorama_test_utility_run.py \\\n", "\\\n", "--app-name {app_name} \\\n", "--code-package-name {code_package_name} \\\n", "--model-package-name {model_package_name} \\\n", "--camera-node-name {camera_node_name} \\\n", "--model-node-name {model_node_name} \\\n", "--model-file-basename {model_file_basename} \\\n", "--video-file {videoname} \\\n", "--py-file ./{app_name}/packages/{account_id}-{code_package_name}-1.0/src/app.py \\\n", "--output-pyplot" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "# Package app\n", "\n", "Updates the app to be deployed with the recent code" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "py_file_name = 'app.py'\n", "panorama_test_utility.update_package_descriptor( app_name, account_id, code_package_name, py_file_name )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Build app with container" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "container_asset_name = 'code_asset'" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture captured_output\n", "\n", "# Building container image.This process takes time (5min ~ 10min)\n", "# FIXME : without %%capture, browser tab crashes because of too much output from the command.\n", "\n", "!cd ./bird_demo_app && panorama-cli build \\\n", " --container-asset-name {container_asset_name} \\\n", " --package-path packages/{account_id}-{code_package_name}-1.0" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "stdout_lines = captured_output.stdout.splitlines()\n", "stderr_lines = captured_output.stderr.splitlines()\n", "print(\" :\")\n", "print(\" :\")\n", "for line in stdout_lines[-30:] + stderr_lines[-30:]:\n", " print(line)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Upload application to Panorama for deploying to devices" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Update camera streams\n", "\n", "In the AWS Panorama console, you can select the camera streams, but programmatically, you need to define the camera stream info for the cameras you are using with the app.\n", "\n", "Open the ```package.json``` in ```packages/-RTSP_STREAM-1.0``` and update the camera username, password and URL. After you have updated your camera credentials, run package-application. You can override this camera stream when you deploy the app." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# This step takes some time, depending on your network environment.\n", "!cd ./bird_demo_app && panorama-cli package-application" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ready for deploying to a device\n", "\n", "Congrats! Your app is now ready to deploy to a device. Next, you can continue in this notebook to deploy the app programmatically or you can go to the Panorama console and deploying using a GUI. The GUI makes it easier to select camera streams and select the devices you want to deploy to. Programmatic deployment is faster to complete and easier to automate.\n", "\n", "# Deploy app to device\n", "\n", "Let's make sure the device we are deploying to is available." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "response = panorama_client.describe_device(\n", " DeviceId= DEVICE_ID\n", ")\n", "\n", "print('You are deploying to Device: {}'.format(response['Name']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Deploy Application" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "response = panorama_test_utility.deploy_app(DEVICE_ID, app_name, role=None)\n", "\n", "application_instance_id = response['ApplicationInstanceId']\n", "\n", "response" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Clean up" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "panorama_test_utility.remove_application( DEVICE_ID, application_instance_id )" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" } }, "nbformat": 4, "nbformat_minor": 5 }