{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object and scene detection using Amazon Rekognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "This notebook provides a walkthrough of [object detection API](https://docs.aws.amazon.com/rekognition/latest/dg/labels.html) in Amazon Rekognition to identify objects.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize dependencies\n",
    "import boto3\n",
    "import botocore\n",
    "from IPython.display import HTML, display, Image as IImage\n",
    "import time\n",
    "\n",
    "# Initialize clients\n",
    "REGION = boto3.session.Session().region_name\n",
    "rekognition = boto3.client('rekognition', REGION)\n",
    "s3 = boto3.client('s3', REGION)\n",
    "\n",
    "%store -r bucket_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detect objects in image\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show image\n",
    "image_name = \"media/cellphone.jpg\"\n",
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': image_name})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Call Rekognition to detect objects in the image\n",
    "\n",
    "Call Amazon Rekognition to detect objects in the image\n",
    "<https://docs.aws.amazon.com/rekognition/latest/dg/labels-detect-labels-image.html>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "detect_labels_response = rekognition.detect_labels(\n",
    "    Image={\n",
    "        'S3Object': {\n",
    "            'Bucket': bucket_name,\n",
    "            'Name': image_name,\n",
    "        }\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review the raw JSON reponse from Rekognition\n",
    "\n",
    "In the JSON response below, you will see Label, detected instances, confidence score and additional information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(detect_labels_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display list of detected unsafe objects\n",
    "\n",
    "You can build your own list of flagged objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flagged_objects = [\"Cell Phone\", \"Electronics\" \"Mobile Phone\", \"Phone\", \"Reading\"]\n",
    "\n",
    "for label in detect_labels_response[\"Labels\"]:\n",
    "    if(label[\"Name\"] in flagged_objects):\n",
    "        print(\"Detected unsafe object:\")\n",
    "        print(\"- {} (Confidence: {})\".format(label[\"Name\"], label[\"Confidence\"]))\n",
    "        print(\"  - Parents: {}\".format(label[\"Parents\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recognize objects in video\n",
    " Object recognition in video is an async operation. \n",
    "https://docs.aws.amazon.com/rekognition/latest/dg/labels-detecting-labels-video.html. \n",
    "\n",
    "- First we start a label detection job which returns a Job Id.\n",
    "- We can then call `get_label_detection` to get the job status and after job is complete, we can get object metadata.\n",
    "- In production use cases, you would usually use StepFunction or SNS topic to get notified when job is complete.\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show video in the player"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "video_name = \"media/objects.mp4\"\n",
    "s3_video_url = s3.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': video_name})\n",
    "\n",
    "video_tag = \"<video controls='controls' autoplay width='640' height='360' name='Video' src='{0}'></video>\".format(s3_video_url)\n",
    "video_ui = \"<table><tr><td style='vertical-align: top'>{}</td></tr></table>\".format(video_tag)\n",
    "display(HTML(video_ui))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Call Rekognition to start a job for object detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_label_detection = rekognition.start_label_detection(\n",
    "    Video={\n",
    "        'S3Object': {\n",
    "            'Bucket': bucket_name,\n",
    "            'Name': video_name,\n",
    "        }\n",
    "    },\n",
    ")\n",
    "\n",
    "labels_job_id = start_label_detection['JobId']\n",
    "display(\"Job Id: {0}\".format(labels_job_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional (Optional) Request Attributes\n",
    "\n",
    "ClientRequestToken, JobTag, MinConfidence, and NotificationChannel:\n",
    "https://docs.aws.amazon.com/rekognition/latest/APIReference/API_StartLabelDetection.html\n",
    "\n",
    "Calling Amazon Rekognition Video Operations:\n",
    "https://docs.aws.amazon.com/rekognition/latest/dg/api-video.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for object detection job to complete\n",
    "\n",
    "In production use cases, you would usually use StepFunction or SNS topic to get notified when job is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_object_detection = rekognition.get_label_detection(\n",
    "    JobId=labels_job_id,\n",
    "    SortBy='TIMESTAMP'\n",
    ")\n",
    "\n",
    "while(get_object_detection['JobStatus'] == 'IN_PROGRESS'):\n",
    "    time.sleep(5)\n",
    "    print('.', end='')\n",
    " \n",
    "    get_object_detection = rekognition.get_label_detection(\n",
    "        JobId=labels_job_id,\n",
    "        SortBy='TIMESTAMP')\n",
    "    \n",
    "display(get_object_detection['JobStatus'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review raw JSON reponse from Rekognition\n",
    "\n",
    "In the JSON response below, you will see list of detected objects and activities.\n",
    "For each detected object, you will see information like Timestamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "display(get_object_detection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display names of recognized objects in the video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "the_objects = {}\n",
    "\n",
    "# Display timestamps and objects detected at that time\n",
    "str_detail = \"Objects detected in video<br>=======================================<br>\"\n",
    "str_overall = \"Objects in the overall video:<br>=======================================<br>\"\n",
    "\n",
    "# Objects detected in each frame\n",
    "for obj in get_object_detection['Labels']:\n",
    "    ts = obj [\"Timestamp\"]\n",
    "    cconfidence = obj['Label'][\"Confidence\"]\n",
    "    oname = obj['Label'][\"Name\"]\n",
    "    \n",
    "    if(oname in flagged_objects):\n",
    "        print(\"Found flagged object at {} ms: {} (Confidence: {})\".format(ts, oname, round(cconfidence,2)))\n",
    "    \n",
    "    str_detail += \"At {} ms: {} (Confidence: {})<br>\".format(ts, oname, round(cconfidence,2))\n",
    "    if oname in the_objects:\n",
    "        cojb = the_objects[oname]\n",
    "        the_objects[oname] = {\"Name\" : oname, \"Count\": 1+cojb[\"Count\"]}\n",
    "    else:\n",
    "        the_objects[oname] = {\"Name\" : oname, \"Count\": 1}\n",
    "\n",
    "# Unique objects detected in video\n",
    "for the_object in the_objects:\n",
    "    str_overall += \"Name: {}, Count: {}<br>\".format(the_object, the_objects[the_object][\"Count\"])\n",
    "\n",
    "# Display results\n",
    "display(HTML(str_overall))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_ui = \"<table><tr><td style='vertical-align: top'>{}</td></tr></table>\".format(str_detail)\n",
    "display(HTML(list_ui))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### References\n",
    "- https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DetectLabels.html\n",
    "- https://docs.aws.amazon.com/rekognition/latest/APIReference/API_StartLabelDetection.html\n",
    "- https://docs.aws.amazon.com/rekognition/latest/APIReference/API_GetLabelDetection.html\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have successfully used Amazon Rekognition to identify specific objects in images and videos."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}