{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This pose_estimation sample application detects people in camera streams, and estimates pose of each person. To achieve this, this sample application uses 2 types of models, 1) people detection model, and 2) pose estimation models.\n",
    "\n",
    "This notebook explains how to build and deploy the application step by step.\n",
    "\n",
    "#### How this application works\n",
    "\n",
    "1. Apply people detection model to camera image.\n",
    "2. Choose up to 4 boxes based on the people detection scores.\n",
    "3. Transform the bounding boxes and sub-images to the input shape of the pose estimation model.\n",
    "4. Apply pose estimation model to the sub images. A pose estimation model with batch size 4 is used.\n",
    "5. Find 17 joint positions for each person, from the output from the pose estimation model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Before you start processing this notebook, some prerequisites need to be completed.\n",
    "\n",
    "* Set up your AWS Panorama Appliance - [middle click to open document](https://docs.aws.amazon.com/panorama/latest/dev/gettingstarted-setup.html)\n",
    "* Create at least one camera stream - [middle click to open document](https://docs.aws.amazon.com/panorama/latest/dev/gettingstarted-setup.html#gettingstarted-setup-camera)\n",
    "* Create an IAM Role for your application - [middle click to open document](https://docs.aws.amazon.com/panorama/latest/dev/permissions-application.html)\n",
    "* Install \"panorama-cli\" tool [middle click to open document](https://docs.aws.amazon.com/panorama/latest/dev/gettingstarted-deploy.html#gettingstarted-deploy-prerequisites)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional libraries\n",
    "\n",
    "This sample requires `mxnet` and `gluoncv` installations to export a CV model from model zoo. If your environment doesn't have them, please open a terminal, and run following commands to install them.\n",
    "\n",
    "**x86 platform**\n",
    "``` bash\n",
    "$ sudo pip3 install mxnet gluoncv\n",
    "```\n",
    "\n",
    "**ARM platform**\n",
    "``` bash\n",
    "$ mkdir install_mxnet\n",
    "$ cd install_mxnet\n",
    "$ sudo apt-get install -y libopencv-dev\n",
    "$ wget https://archive.apache.org/dist/incubator/mxnet/1.8.0/apache-mxnet-src-1.8.0-incubating.tar.gz\n",
    "$ tar xvzf apache-mxnet-src-1.8.0-incubating.tar.gz\n",
    "$ cd apache-mxnet-src-1.8.0-incubating\n",
    "$ make -j8\n",
    "$ cd python\n",
    "$ sudo python3 setup.py install\n",
    "$ sudo pip3 install gluoncv\n",
    "```\n",
    "\n",
    "And restart the kernel "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Import libraries and define configurations\n",
    "\n",
    "First step is to import all libraries needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import time\n",
    "import tarfile\n",
    "import json\n",
    "import uuid\n",
    "\n",
    "import boto3\n",
    "import gluoncv\n",
    "\n",
    "sys.path.insert( 0, os.path.abspath( \"../common/test_utility\" ) )\n",
    "import panorama_test_utility\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to specify some information specific to your environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "account_id = input( \"Your AWS account ID (format : 12 digits)\" ).strip()\n",
    "region_name = input(\"Region name (e.g. us-east-1)\").strip()\n",
    "s3_bucket = input( \"Your S3 bucket for model compilation\" ).strip()\n",
    "\n",
    "# Following configurations are required when you use real hardware, \n",
    "# thus can be any dummy strings when you use only Test Utility.\n",
    "device_id = input(\"Device Id (format : device-*)\").strip()\n",
    "runtime_role_arn = input(\"IAM Role ARN for the application on the device\").strip()\n",
    "data_source_name = input( \"Your data source (Camera) name\" ).strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Import application\n",
    "\n",
    "With \"panorama-cli import-application\" command, replacing placeholder information in application files. This step essentially replace placeholder (\"123456789012\") with your aws account id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd ./pose_estimation_app/ && panorama-cli import-application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Export models from GluonCV model zoo\n",
    "\n",
    "Exporting models from GluonCV's model zoo. This sample uses \"yolo3_mobilenet1.0_coco\" for people detection, and \"simple_pose_resnet152_v1d\" for pose estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def export_model_and_create_targz( prefix, name, model ):\n",
    "    os.makedirs( prefix, exist_ok=True )\n",
    "    gluoncv.utils.export_block( os.path.join( prefix, name ), model, preprocess=False, layout=\"CHW\" )\n",
    "\n",
    "    tar_gz_filename = f\"{prefix}/{name}.tar.gz\"\n",
    "    with tarfile.open( tar_gz_filename, \"w:gz\" ) as tgz:\n",
    "        tgz.add( f\"{prefix}/{name}-symbol.json\", f\"{name}-symbol.json\" )\n",
    "        tgz.add( f\"{prefix}/{name}-0000.params\", f\"{name}-0000.params\" )\n",
    "        \n",
    "    print( f\"Exported : {tar_gz_filename}\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export object detection model. Reset the classes for human detection only.\n",
    "people_detection_model = gluoncv.model_zoo.get_model('yolo3_mobilenet1.0_coco', pretrained=True)\n",
    "people_detection_model.reset_class([\"person\"], reuse_weights=['person'])\n",
    "export_model_and_create_targz( \"models\", \"yolo3_mobilenet1.0_coco_person\", people_detection_model )\n",
    "\n",
    "# Export pose estimation model.\n",
    "pose_estimation_model = gluoncv.model_zoo.get_model('simple_pose_resnet152_v1d', pretrained=True)\n",
    "export_model_and_create_targz( \"models\", \"simple_pose_resnet152_v1d\", pose_estimation_model )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Add exported model files in the model package\n",
    "\n",
    "With \"panorama-cli add-raw-model\" command, Adding exported models into a model package.\n",
    "\n",
    "Please note that we use 4 variations of simple_pose_resnet152_v1d models with different input data shapes ( [1, 3, 256, 192] ~ [4, 3, 256, 192] ), so there are 5 models in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd ./pose_estimation_app/ && panorama-cli add-raw-model \\\n",
    "    --model-asset-name people_detection_model \\\n",
    "    --model-local-path ../models/yolo3_mobilenet1.0_coco_person.tar.gz \\\n",
    "    --descriptor-path packages/{account_id}-pose_estimation_models-1.0/descriptor_people_detection.json \\\n",
    "    --packages-path packages/{account_id}-pose_estimation_models-1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_batch_images = 4\n",
    "!cd ./pose_estimation_app/ && panorama-cli add-raw-model \\\n",
    "    --model-asset-name pose_estimation_model_{num_batch_images} \\\n",
    "    --model-local-path ../models/simple_pose_resnet152_v1d.tar.gz \\\n",
    "    --descriptor-path packages/{account_id}-pose_estimation_models-1.0/descriptor_pose_estimation_{num_batch_images}.json \\\n",
    "    --packages-path packages/{account_id}-pose_estimation_models-1.0\n",
    "    \n",
    "    print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare business logic\n",
    "\n",
    "#### Preview python source code\n",
    "Next step is to build a business logic container. This application's business logic consists of single python source code. Let's preview it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "panorama_test_utility.preview_text_file( f\"./pose_estimation_app/packages/{account_id}-pose_estimation_code-1.0/src/app.py\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test run the business logic with test-utility\n",
    "\n",
    "Let's run the application with Test Utility, with following steps.\n",
    "\n",
    "1. Compile models with Test Utility **Compile** command.\n",
    "2. Run application with Test Utility **Run** command, and check if it runs as expected.\n",
    "\n",
    "To iterate the development, you can edit the python source code with your prefered text editor (or even within Jupyter environment), and re-run the application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the models to run with test-utility.\n",
    "# Specifying 5 sets of model related arguments to compile 5 models.\n",
    "# This step takes 40 mins ~ 60 mins.\n",
    "\n",
    "people_detection_model_data_shape = '{\"data\":[1,3,480,600]}'\n",
    "\n",
    "pose_estimation_model_data_shapes = '{\"data\":[4,3,256,192]}'\n",
    "\n",
    "%run ../common/test_utility/panorama_test_utility_compile.py \\\n",
    "\\\n",
    "--s3-model-location s3://{s3_bucket}/pose_estimation_app \\\n",
    "\\\n",
    "--model-node-name people_detection_model \\\n",
    "--model-file-basename ./models/yolo3_mobilenet1.0_coco_person \\\n",
    "--model-data-shape '{people_detection_model_data_shape}' \\\n",
    "--model-framework MXNET \\\n",
    "\\\n",
    "--model-node-name pose_estimation_model_4 \\\n",
    "--model-file-basename ./models/simple_pose_resnet152_v1d \\\n",
    "--model-data-shape '{pose_estimation_model_data_shapes}' \\\n",
    "--model-framework MXNET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Run the application with test-utility.\n",
    "#\n",
    "# As '--output-screenshots' option is specified, this command simulates HDMI output by generating sequentially numbered screenshot files.\n",
    "# You can find screenshot files under ./screenshot directory.\n",
    "\n",
    "video_filepath = \"../common/test_utility/videos/TownCentreXVID.avi\"\n",
    "\n",
    "%run ../common/test_utility/panorama_test_utility_run.py \\\n",
    "--app-name pose_estimation_app \\\n",
    "--code-package-name pose_estimation_code \\\n",
    "--model-package-name pose_estimation_models \\\n",
    "--camera-node-name abstract_rtsp_media_source \\\n",
    "\\\n",
    "--model-node-name people_detection_model \\\n",
    "--model-file-basename ./models/yolo3_mobilenet1.0_coco_person \\\n",
    "\\\n",
    "--model-node-name pose_estimation_model_4 \\\n",
    "--model-file-basename ./models/simple_pose_resnet152_v1d \\\n",
    "\\\n",
    "--video-file {video_filepath} \\\n",
    "--video-start 0 \\\n",
    "--video-stop 30 \\\n",
    "--video-step 1 \\\n",
    "\\\n",
    "--output-screenshots ./screenshot/%Y%m%d-%H%M%S \\\n",
    "\\\n",
    "--py-file ./pose_estimation_app/packages/{account_id}-pose_estimation_code-1.0/src/app.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import IPython\n",
    "\n",
    "latest_screenshot_dirname = sorted( glob.glob( \"./screenshot/*\" ) )[-1]\n",
    "screenshot_filename = sorted( glob.glob( f\"{latest_screenshot_dirname}/*.png\" ) )[-1]\n",
    "\n",
    "print(screenshot_filename)\n",
    "IPython.display.Image( filename = screenshot_filename )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build application logic container\n",
    "\n",
    "With \"panorama-cli build-container\" command, building a container image, and add it into the \"pose_estimation_code\" package.\n",
    "\n",
    "This step takes long time (5~10 mins), and because it is using %%capture magic command, you don't see progress during the process. Please wait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture captured_output\n",
    "# FIXME : without %%capture, browser tab crashes because of too much output from the command.\n",
    "\n",
    "!cd ./pose_estimation_app && panorama-cli build-container \\\n",
    "    --container-asset-name code \\\n",
    "    --package-path packages/{account_id}-pose_estimation_code-1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stdout_lines = captured_output.stdout.splitlines()\n",
    "stderr_lines = captured_output.stderr.splitlines()\n",
    "print(\"     :\")\n",
    "print(\"     :\")\n",
    "for line in stdout_lines[-30:] + stderr_lines[-30:]:\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Package application (upload locally prepared packages onto Cloud)\n",
    "\n",
    "Now you have prepared both model packages and code package locally. Let's upload those packages to the cloud with \"panorama-cli package-application\" command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd ./pose_estimation_app && panorama-cli package-application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deploy application to the device programatically\n",
    "\n",
    "Once you uploaded the packages to the cloud, you can create an application instance on your device. You need to specify a manifest file, and optionally an override-manifest file. In this sample app, \"panorama::abstract_rtsp_media_source\" is used as a data source node, and it has to be overridden by override manifest file, so using both files.\n",
    "\n",
    "You can manually create override-manufest file, but in this sample notebook, we create it programmatically from template file and the Data Source name you input at the beginning.\n",
    "\n",
    "#### Advanced topic\n",
    "If you have multiple cameras, you can include them in the \"override.json\". Please refer to \"override_template_multi_cameras.json\" how to include multiple cameras in the override maifest file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read template file\n",
    "with open( \"./pose_estimation_app/graphs/pose_estimation_app/override_template.json\", \"r\" ) as fd:\n",
    "    data = fd.read()\n",
    "\n",
    "# Replace placeholders\n",
    "data = data.replace( \"{YourAwsAccountId}\", account_id )\n",
    "data = data.replace( \"{YourCameraName}\", data_source_name )\n",
    "\n",
    "# Write override-manifest file\n",
    "with open( \"./pose_estimation_app/graphs/pose_estimation_app/override.json\", \"w\" ) as fd:\n",
    "    fd.write(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Preview manifest file(\"graph.json\"), and override-manifest file(\"override.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "panorama_test_utility.preview_text_file( \"./pose_estimation_app/graphs/pose_estimation_app/graph.json\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "panorama_test_utility.preview_text_file( \"./pose_estimation_app/graphs/pose_estimation_app/override.json\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the app using the manifest files\n",
    "\n",
    "In order to create an application instance, this notebook uses boto3's \"panorama\" client and create_application_instance() API. (It is also possible to use \"aws panorama create-application-instance\" command instead.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a boto3 client to access Panorama service\n",
    "# FIXME : not using AWS_REGION here, because panorama-cli uses only default region currently.\n",
    "panorama_client = boto3.client(\"panorama\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deploy_application( application_name, manifest_filename, override_filename ):\n",
    "\n",
    "    def get_payload_from_json( filename ):\n",
    "        with open( filename ) as fd:\n",
    "            \n",
    "            s = fd.read()\n",
    "            \n",
    "            assert \"{Your\" not in s, \"Please replace {YourAwsAccountId} and {YourCameraName} in [%s] with your AWS account id\" % (filename)\n",
    "            \n",
    "            # validating JSON format and making it compact, by loading and dumping, \n",
    "            payload = json.dumps(json.loads(s))\n",
    "            \n",
    "            return payload\n",
    "\n",
    "    manifest_payload = get_payload_from_json( manifest_filename )\n",
    "    override_payload = get_payload_from_json( override_filename )\n",
    "    \n",
    "    response = panorama_client.create_application_instance(\n",
    "        Name = application_name,\n",
    "        RuntimeRoleArn = runtime_role_arn,\n",
    "        DefaultRuntimeContextDevice = device_id,\n",
    "        ManifestPayload = {\"PayloadData\":manifest_payload},\n",
    "        ManifestOverridesPayload = {\"PayloadData\":override_payload},\n",
    "    )\n",
    "        \n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "application_name = \"pose_estimation_notebook_\" + str(uuid.uuid4())[:8]\n",
    "\n",
    "response = deploy_application(\n",
    "    application_name = application_name,\n",
    "    manifest_filename = \"./pose_estimation_app/graphs/pose_estimation_app/graph.json\",\n",
    "    override_filename = \"./pose_estimation_app/graphs/pose_estimation_app/override.json\"\n",
    ")\n",
    "\n",
    "application_instance_id = response[\"ApplicationInstanceId\"]\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "application_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Wait for deployment completion\n",
    "\n",
    "Application instance creation has been triggered. This notebook checks the progress by calling describe_application_instance() API periodically. Please confirm that you see \"DEPLOYMENT_SUCCEEDED\" status at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wait_deployment( application_instance_id ):\n",
    "    \n",
    "    progress_dots = panorama_test_utility.ProgressDots()    \n",
    "    while True:\n",
    "        app = panorama_client.describe_application_instance( ApplicationInstanceId = application_instance_id )\n",
    "        progress_dots.update_status( \"%s (%s)\" % (app[\"Status\"], app[\"StatusDescription\"]) )\n",
    "        if app[\"Status\"] not in ( \"DEPLOYMENT_PENDING\", \"DEPLOYMENT_REQUESTED\", \"DEPLOYMENT_IN_PROGRESS\" ):\n",
    "            break\n",
    "        time.sleep(60)\n",
    "\n",
    "wait_deployment( application_instance_id )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Visit CloudWatch Logs to check logs from the application instance\n",
    "\n",
    "If you saw \"DEPLOYMENT_SUCCEEDED\" status, the application started to run on your device. Application logs are uploaded to CloudWatch Logs. Let's get the URL of CloudWatch Logs management console. \"console_output\" is the log stream your Python code's stdout/stderr are redirected to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logs_url = panorama_test_utility.get_logs_url( region_name, device_id, application_instance_id )\n",
    "print( \"CloudWatch Logs URL :\" )\n",
    "print( logs_url )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove application instance\n",
    "\n",
    "Once you confirmed the successful run of application, let's remove the application instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = input(\"Remove application? [yN]\")\n",
    "if answer.lower()==\"y\":\n",
    "    panorama_test_utility.remove_application( device_id, application_instance_id )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "If you want to customize this sample application, you can follow these steps:\n",
    "\n",
    "1. Edit \"pose_estimation_app/packages/{account_id}-pose_estimation_code-1.0/src/app.py\". You can use your preferred text editor.\n",
    "2. Test-run the updated script with Test Utility [from here](#Test-run-the-business-logic-with-test-utility). Confirm that the application runs as expected.\n",
    "3. Follow the steps for the real device ([build new container image](#Build-application-logic-container), [upload the code package](#Package-application-(upload-locally-prepared-packages-onto-Cloud)), and [deploy to the device](#Deploy-the-app-using-the-manifest-files) ). Confirm that the application runs as expected on the device as well.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "0bf254f4bade2f1c26977f6424deaa54afd031ced29e33743fcf1b047c1a16ff"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}