{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hotdog or Not HotDog\n",
    "\n",
    "Welcome to this SageMaker Notebook! This is an entirely managed notebook service that you can use to create and edit machine learning models. We will be using it today to create a binary image classification model using the Apache MXNet deep learning framework. We will then learn how to delpoy this model onto our DeepLens device.\n",
    "\n",
    "In this notebook we will be to using MXNet's Gluon interface, to download and edit a pre-trained ImageNet model and transform it into binary classifier, which we can use to differentiate between hot dogs and not hot dogs.\n",
    "\n",
    "### Setup\n",
    "\n",
    "Before we start, make sure the kernel in the the notebook is set to the correct one, `condamxnet3.6` which has all the dependencies we will need for this tutorial already installed.\n",
    "\n",
    "First we'll start by importing a bunch of packages into the notebook that you'll need later and installing any required packages that are missing into our notebook kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching package metadata ...........\n",
      "Solving package specifications: .\n",
      "\n",
      "Package plan for installation in environment /home/ec2-user/anaconda3/envs/mxnet_p36:\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "    scikit-image: 0.13.0-py36had3c07a_1 --> 0.13.1-py36ha4a0841_0\n",
      "\n",
      "The following packages will be DOWNGRADED:\n",
      "\n",
      "    networkx:     2.0-py36h7e96fb8_0    --> 1.11-py36hfb3574a_0  \n",
      "\n",
      "Proceed ([y]/n)? \n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "conda install scikit-image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/matplotlib/colors.py:680: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.\n",
      "  not cbook.is_string_like(colors[0]):\n",
      "/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py:46: DeprecationWarning: OpenSSL.rand is deprecated - you should use os.urandom instead\n",
      "  import OpenSSL.SSL\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "import logging\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "import os\n",
    "import time\n",
    "from collections import OrderedDict\n",
    "import skimage.io as io\n",
    "import numpy as np\n",
    "\n",
    "import mxnet as mx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "The model we will be downloading and editing is [SqueezeNet](https://arxiv.org/abs/1602.07360), an extremely efficient image classification model that achived 2012 State of the Art accuracy on the popular [ImageNet](http://www.image-net.org/challenges/LSVRC/), image classification challenge. SqueezeNet is just a convolutional neural network, with an architecture chosen to have a small number of parameters and to require a minimal amount of computation. It's especially popular for folks that need to run CNNs on low-powered devices like cell phones and other internet-of-things devices, such as DeepLens. The MXNet Deep Learning framework offers squeezenet v1.0 and v1.1 that are pretrained on ImageNet through it's model Zoo.\n",
    "\n",
    "## Pulling the pre-trained model\n",
    "The MXNet model zoo  gives us convenient access to a number of popular models,\n",
    "both their architectures and their pretrained parameters.\n",
    "Let's download SqueezeNet right now with just a few lines of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mxnet.gluon import nn\n",
    "from mxnet.gluon.model_zoo import vision as models\n",
    "\n",
    "# get pretrained squeezenet\n",
    "net = models.squeezenet1_1(pretrained=True, prefix='deep_dog_')\n",
    "# hot dog happens to be a class in imagenet.\n",
    "# we can reuse the weight for that class for better performance\n",
    "# here's the index for that class for later use\n",
    "imagenet_hotdog_index = 713"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DeepDog Net\n",
    "\n",
    "In vision networks its common that the first set of layers learns the task of recognizing edges, curves and other important visual features of the input image. We call this feature extraction, and once the abstract features are extracted we can leverage a much simpler model to classify images using these features.\n",
    "\n",
    "We will use the feature extractor from the pretrained squeezenet (every layer except the last one) to build our own classifier for hotdogs. Conveniently, the MXNet model zoo handles the decaptiation for us. All we have to do is specify the number out of output classes in our new task, which we do via the keyword argument `classes=2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SqueezeNet(\n",
      "  (features): HybridSequential(\n",
      "    (0): Conv2D(64, kernel_size=(3, 3), stride=(2, 2))\n",
      "    (1): Activation(relu)\n",
      "    (2): MaxPool2D(size=(3, 3), stride=(2, 2), padding=(0, 0), ceil_mode=True)\n",
      "    (3): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(16, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(64, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (4): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(16, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(64, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (5): MaxPool2D(size=(3, 3), stride=(2, 2), padding=(0, 0), ceil_mode=True)\n",
      "    (6): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(32, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(128, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (7): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(32, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(128, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (8): MaxPool2D(size=(3, 3), stride=(2, 2), padding=(0, 0), ceil_mode=True)\n",
      "    (9): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(48, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(192, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (10): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(48, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(192, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (11): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(64, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(256, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "    (12): HybridSequential(\n",
      "      (0): HybridSequential(\n",
      "        (0): Conv2D(64, kernel_size=(1, 1), stride=(1, 1))\n",
      "        (1): Activation(relu)\n",
      "      )\n",
      "      (1): HybridConcurrent(\n",
      "        (0): HybridSequential(\n",
      "          (0): Conv2D(256, kernel_size=(1, 1), stride=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "        (1): HybridSequential(\n",
      "          (0): Conv2D(256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "          (1): Activation(relu)\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (classifier): HybridSequential(\n",
      "    (0): Dropout(p = 0.5)\n",
      "    (1): Conv2D(2, kernel_size=(1, 1), stride=(1, 1))\n",
      "    (2): Activation(relu)\n",
      "    (3): AvgPool2D(size=(13, 13), stride=(13, 13), padding=(0, 0), ceil_mode=False)\n",
      "    (4): Flatten\n",
      "  )\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "deep_dog_net = models.squeezenet1_1(prefix='deep_dog_', classes=2)\n",
    "deep_dog_net.collect_params().initialize()\n",
    "deep_dog_net.features = net.features\n",
    "\n",
    "# Lets take a look at what this network looks like\n",
    "print(deep_dog_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network can already be used for prediction. However, since it hasn't been finetuned yet so the network performance could not be optimal.\n",
    "\n",
    "Let's test it out by defining a prediction function to feed a local image into the network and get the predicted output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from skimage.color import rgba2rgb\n",
    "\n",
    "def classify_hotdog(net, url):\n",
    "    I = io.imread(url)\n",
    "    if I.shape[2] == 4:\n",
    "        I = rgba2rgb(I)\n",
    "    image = mx.nd.array(I).astype(np.uint8)\n",
    "    image = mx.image.resize_short(image, 256)\n",
    "    image, _ = mx.image.center_crop(image, (224, 224))\n",
    "    image = mx.image.color_normalize(image.astype(np.float32)/255,\n",
    "                                     mean=mx.nd.array([0.485, 0.456, 0.406]),\n",
    "                                     std=mx.nd.array([0.229, 0.224, 0.225]))\n",
    "    image = mx.nd.transpose(image.astype('float32'), (2,1,0))\n",
    "    image = mx.nd.expand_dims(image, axis=0)\n",
    "    out = mx.nd.SoftmaxActivation(net(image))\n",
    "    print('Probabilities are: '+str(out[0].asnumpy()))\n",
    "    result = np.argmax(out.asnumpy())\n",
    "    outstring = ['Not hotdog!', 'Hotdog!']\n",
    "    print(outstring[result])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets download a hot dog image and an image of another object to our local directory to test this model on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "--2017-11-24 08:32:34--  http://www.wienerschnitzel.com/wp-content/uploads/2014/10/hotdog_mustard-main.jpg\n",
      "Resolving www.wienerschnitzel.com (www.wienerschnitzel.com)... 104.198.109.247\n",
      "Connecting to www.wienerschnitzel.com (www.wienerschnitzel.com)|104.198.109.247|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 22917 (22K) [image/jpeg]\n",
      "Saving to: ‘hotdog_mustard-main.jpg.1’\n",
      "\n",
      "     0K .......... .......... ..                              100%  358K=0.06s\n",
      "\n",
      "2017-11-24 08:32:34 (358 KB/s) - ‘hotdog_mustard-main.jpg.1’ saved [22917/22917]\n",
      "\n",
      "--2017-11-24 08:32:34--  https://www.what-dog.net/Images/faces2/scroll001.jpg\n",
      "Resolving www.what-dog.net (www.what-dog.net)... 191.237.47.20\n",
      "Connecting to www.what-dog.net (www.what-dog.net)|191.237.47.20|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 48316 (47K) [image/jpeg]\n",
      "Saving to: ‘scroll001.jpg’\n",
      "\n",
      "     0K .......... .......... .......... .......... .......   100% 8.58M=0.005s\n",
      "\n",
      "2017-11-24 08:32:34 (8.58 MB/s) - ‘scroll001.jpg’ saved [48316/48316]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "wget http://www.wienerschnitzel.com/wp-content/uploads/2014/10/hotdog_mustard-main.jpg\n",
    "wget https://www.what-dog.net/Images/faces2/scroll001.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probabilities are: [ 0.66635329  0.33364674]\n",
      "Not hotdog!\n",
      "Probabilities are: [ 0.48589313  0.51410687]\n",
      "Hotdog!\n"
     ]
    }
   ],
   "source": [
    "# To make the defined network run quickly we usually hybridize it first. \n",
    "# This also allows us to serialize and export our model\n",
    "deep_dog_net.hybridize()\n",
    "\n",
    "# Let's run the classification on our tow downloaded images to see what our model comes up with\n",
    "classify_hotdog(deep_dog_net, './hotdog_mustard-main.jpg') # check for hotdog\n",
    "classify_hotdog(deep_dog_net, './scroll001.jpg') # check for not-hotdog"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "deep_dog_net.export('hotdog_or_not_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions are a bit off so we can download a set of new parameters for the model that we have pre-optimized through a \"fine tuning\" process, where we retrained the model with images of hotdogs and not hotdogs. We can then apply these new parameters to our model to make it even more accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:downloaded https://apache-mxnet.s3-accelerate.amazonaws.com/gluon/models/deep-dog-5a342a6f.params into deep-dog-5a342a6f.params successfully\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probabilities are: [ 0.37135434  0.62864566]\n",
      "Hotdog!\n",
      "Probabilities are: [ 0.99881637  0.00118361]\n",
      "Not hotdog!\n"
     ]
    }
   ],
   "source": [
    "from mxnet.test_utils import download\n",
    "\n",
    "download('https://apache-mxnet.s3-accelerate.amazonaws.com/gluon/models/deep-dog-5a342a6f.params',\n",
    "         overwrite=True)\n",
    "deep_dog_net.load_params('deep-dog-5a342a6f.params', mx.cpu())\n",
    "deep_dog_net.hybridize()\n",
    "classify_hotdog(deep_dog_net, './hotdog_mustard-main.jpg')\n",
    "classify_hotdog(deep_dog_net, './scroll001.jpg')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions seem reasonable, so we can export this as a serialized model to our local dirctory. This is a simple one line command, which produces a set of two files: a json file holding the network architecture, and a params file holding the parameters the network learned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "deep_dog_net.export('hotdog_or_not_model_v2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's push this serialized model to S3, where we can then optimize it for our DeepLense device and then push it down onto our device for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): sts.amazonaws.com\n",
      "INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:iam::622803848910:role/SageMaker_role_IM\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "s3.Object(bucket_name='sagemaker-test1', key='hotdog_or_not_model-0000.params')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import boto3\n",
    "import re\n",
    "\n",
    "assumed_role = boto3.client('sts').get_caller_identity()['Arn']\n",
    "s3_access_role = re.sub(r'^(.+)sts::(\\d+):assumed-role/(.+?)/.*$', r'\\1iam::\\2:role/\\3', assumed_role)\n",
    "print(s3_access_role)\n",
    "s3 = boto3.resource('s3')\n",
    "bucket= 'your s3 bucket name here' \n",
    "\n",
    "json = open('hotdog_or_not_model-symbol.json', 'rb')\n",
    "params = open('hotdog_or_not_model-0000.params', 'rb')\n",
    "s3.Bucket(bucket).put_object(Key='test/hotdog_or_not_model-symbol.json', Body=json)\n",
    "s3.Bucket(bucket).put_object(Key='test/hotdog_or_not_model-0000.params', Body=params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Environment (conda_mxnet_p36)",
   "language": "python",
   "name": "conda_mxnet_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}