{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Building your own TensorFlow container\n", "\n", "With Amazon SageMaker, you can package your own algorithms that can then be trained and deployed in the SageMaker environment. This notebook guides you through an example using TensorFlow that shows you how to build a Docker container for SageMaker and use it for training and inference.\n", "\n", "By packaging an algorithm in a container, you can bring almost any code to the Amazon SageMaker environment, regardless of programming language, environment, framework, or dependencies. \n", "\n", "1. [Building your own TensorFlow container](#Building-your-own-tensorflow-container)\n", " 1. [When should I build my own algorithm container?](#When-should-I-build-my-own-algorithm-container?)\n", " 1. [Permissions](#Permissions)\n", " 1. [The example](#The-example)\n", " 1. [The presentation](#The-presentation)\n", "1. [Part 1: Packaging and Uploading your Algorithm for use with Amazon SageMaker](#Part-1:-Packaging-and-Uploading-your-Algorithm-for-use-with-Amazon-SageMaker)\n", " 1. [An overview of Docker](#An-overview-of-Docker)\n", " 1. [How Amazon SageMaker runs your Docker container](#How-Amazon-SageMaker-runs-your-Docker-container)\n", " 1. [Running your container during training](#Running-your-container-during-training)\n", " 1. [The input](#The-input)\n", " 1. [The output](#The-output)\n", " 1. [Running your container during hosting](#Running-your-container-during-hosting)\n", " 1. [The parts of the sample container](#The-parts-of-the-sample-container)\n", " 1. [The Dockerfile](#The-Dockerfile)\n", " 1. [Building and registering the container](#Building-and-registering-the-container)\n", " 1. [Testing your algorithm on your local machine](#Testing-your-algorithm-on-your-local-machine)\n", "1. [Part 2: Training and Hosting your Algorithm in Amazon SageMaker](#Part-2:-Training-and-Hosting-your-Algorithm-in-Amazon-SageMaker)\n", " 1. [Set up the environment](#Set-up-the-environment)\n", " 1. [Create the session](#Create-the-session)\n", " 1. [Upload the data for training](#Upload-the-data-for-training)\n", " 1. [Training On SageMaker](#Training-on-SageMaker)\n", " 1. [Optional cleanup](#Optional-cleanup) \n", "1. [Reference](#Reference)\n", "\n", "_or_ I'm impatient, just [let me see the code](#The-Dockerfile)!\n", "\n", "## When should I build my own algorithm container?\n", "\n", "You may not need to create a container to bring your own code to Amazon SageMaker. When you are using a framework such as Apache MXNet or TensorFlow that has direct support in SageMaker, you can simply supply the Python code that implements your algorithm using the SDK entry points for that framework. This set of supported frameworks is regularly added to, so you should check the current list to determine whether your algorithm is written in one of these common machine learning environments.\n", "\n", "Even if there is direct SDK support for your environment or framework, you may find it more effective to build your own container. If the code that implements your algorithm is quite complex or you need special additions to the framework, building your own container may be the right choice.\n", "\n", "Some of the reasons to build an already supported framework container are:\n", "1. A specific version isn't supported.\n", "2. Configure and install your dependencies and environment.\n", "3. Use a different training/hosting solution than provided.\n", "\n", "This walkthrough shows that it is quite straightforward to build your own container. So you can still use SageMaker even if your use case is not covered by the deep learning containers that we've built for you.\n", "\n", "## Permissions\n", "\n", "Running this notebook requires permissions in addition to the normal `SageMakerFullAccess` permissions. This is because it creates new repositories in Amazon ECR. The easiest way to add these permissions is simply to add the managed policy `AmazonEC2ContainerRegistryFullAccess` to the role that you used to start your notebook instance. There's no need to restart your notebook instance when you do this, the new permissions will be available immediately.\n", "\n", "## The example\n", "\n", "In this example we show how to package a custom TensorFlow container with a Python example which works with the CIFAR-10 dataset and uses TensorFlow Serving for inference. However, different inference solutions other than TensorFlow Serving can be used by modifying the docker container.\n", "\n", "In this example, we use a single image to support training and hosting. This simplifies the procedure because we only need to manage one image for both tasks. Sometimes you may want separate images for training and hosting because they have different requirements. In this case, separate the parts discussed below into separate Dockerfiles and build two images. Choosing whether to use a single image or two images is a matter of what is most convenient for you to develop and manage.\n", "\n", "If you're only using Amazon SageMaker for training or hosting, but not both, only the functionality used needs to be built into your container.\n", "\n", "[CIFAR-10]: http://www.cs.toronto.edu/~kriz/cifar.html\n", "\n", "## The presentation\n", "\n", "This presentation is divided into two parts: _building_ the container and _using_ the container." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Part 1: Packaging and Uploading your Algorithm for use with Amazon SageMaker\n", "\n", "### An overview of Docker\n", "\n", "If you're familiar with Docker already, you can skip ahead to the next section.\n", "\n", "For many data scientists, Docker containers are a new technology. But they are not difficult and can significantly simply the deployment of your software packages. \n", "\n", "Docker provides a simple way to package arbitrary code into an _image_ that is totally self-contained. Once you have an image, you can use Docker to run a _container_ based on that image. Running a container is just like running a program on the machine except that the container creates a fully self-contained environment for the program to run. Containers are isolated from each other and from the host environment, so the way your program is set up is the way it runs, no matter where you run it.\n", "\n", "Docker is more powerful than environment managers like conda or virtualenv because (a) it is completely language independent and (b) it comprises your whole operating environment, including startup commands, and environment variable.\n", "\n", "A Docker container is like a virtual machine, but it is much lighter weight. For example, a program running in a container can start in less than a second and many containers can run simultaneously on the same physical or virtual machine instance.\n", "\n", "Docker uses a simple file called a `Dockerfile` to specify how the image is assembled. An example is provided below. You can build your Docker images based on Docker images built by yourself or by others, which can simplify things quite a bit.\n", "\n", "Docker has become very popular in programming and devops communities due to its flexibility and its well-defined specification of how code can be run in its containers. It is the underpinning of many services built in the past few years, such as [Amazon ECS].\n", "\n", "Amazon SageMaker uses Docker to allow users to train and deploy arbitrary algorithms.\n", "\n", "In Amazon SageMaker, Docker containers are invoked in a one way for training and another, slightly different, way for hosting. The following sections outline how to build containers for the SageMaker environment.\n", "\n", "Some helpful links:\n", "\n", "* [Docker home page](http://www.docker.com)\n", "* [Getting started with Docker](https://docs.docker.com/get-started/)\n", "* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/)\n", "* [`docker run` reference](https://docs.docker.com/engine/reference/run/)\n", "\n", "[Amazon ECS]: https://aws.amazon.com/ecs/\n", "\n", "### How Amazon SageMaker runs your Docker container\n", "\n", "Because you can run the same image in training or hosting, Amazon SageMaker runs your container with the argument `train` or `serve`. How your container processes this argument depends on the container.\n", "\n", "* In this example, we don't define an `ENTRYPOINT` in the Dockerfile so Docker runs the command [`train` at training time](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html) and [`serve` at serving time](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html). In this example, we define these as executable Python scripts, but they could be any program that we want to start in that environment.\n", "* If you specify a program as an `ENTRYPOINT` in the Dockerfile, that program will be run at startup and its first argument will be `train` or `serve`. The program can then look at that argument and decide what to do.\n", "* If you are building separate containers for training and hosting (or building only for one or the other), you can define a program as an `ENTRYPOINT` in the Dockerfile and ignore (or verify) the first argument passed in. \n", "\n", "#### Running your container during training\n", "\n", "When Amazon SageMaker runs training, your `train` script is run, as in a regular Python program. A number of files are laid out for your use, under the `/opt/ml` directory:\n", "\n", " /opt/ml\n", " |-- input\n", " | |-- config\n", " | | |-- hyperparameters.json\n", " | | `-- resourceConfig.json\n", " | `-- data\n", " | `-- \n", " | `-- \n", " |-- model\n", " | `-- \n", " `-- output\n", " `-- failure\n", "\n", "##### The input\n", "\n", "* `/opt/ml/input/config` contains information to control how your program runs. `hyperparameters.json` is a JSON-formatted dictionary of hyperparameter names to values. These values are always strings, so you may need to convert them. `resourceConfig.json` is a JSON-formatted file that describes the network layout used for distributed training.\n", "* `/opt/ml/input/data//` (for File mode) contains the input data for that channel. The channels are created based on the call to CreateTrainingJob but it's generally important that channels match algorithm expectations. The files for each channel are copied from S3 to this directory, preserving the tree structure indicated by the S3 key structure. \n", "* `/opt/ml/input/data/_` (for Pipe mode) is the pipe for a given epoch. Epochs start at zero and go up by one each time you read them. There is no limit to the number of epochs that you can run, but you must close each pipe before reading the next epoch.\n", "\n", "##### The output\n", "\n", "* `/opt/ml/model/` is the directory where you write the model that your algorithm generates. Your model can be in any format that you want. It can be a single file or a whole directory tree. SageMaker packages any files in this directory into a compressed tar archive file. This file is made available at the S3 location returned in the `DescribeTrainingJob` result.\n", "* `/opt/ml/output` is a directory where the algorithm can write a file `failure` that describes why the job failed. The contents of this file are returned in the `FailureReason` field of the `DescribeTrainingJob` result. For jobs that succeed, there is no reason to write this file as it is ignored.\n", "\n", "#### Running your container during hosting\n", "\n", "Hosting has a very different model than training because hosting is reponding to inference requests that come in via HTTP. In this example, we use [TensorFlow Serving](https://www.tensorflow.org/serving/), however the hosting solution can be customized. One example is the [Python serving stack within the scikit learn example](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb).\n", "\n", "Amazon SageMaker uses two URLs in the container:\n", "\n", "* `/ping` receives `GET` requests from the infrastructure. Your program returns 200 if the container is up and accepting requests.\n", "* `/invocations` is the endpoint that receives client inference `POST` requests. The format of the request and the response is up to the algorithm. If the client supplied `ContentType` and `Accept` headers, these are passed in as well. \n", "\n", "The container has the model files in the same place that they were written to during training:\n", "\n", " /opt/ml\n", " `-- model\n", " `-- \n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The parts of the sample container\n", "\n", "The `container` directory has all the components you need to package the sample algorithm for Amazon SageMager:\n", "\n", " .\n", " |-- Dockerfile\n", " |-- build_and_push.sh\n", " `-- cifar10\n", " |-- cifar10.py\n", " |-- nginx.conf\n", " |-- serve\n", " `-- train\n", "\n", "Let's discuss each of these in turn:\n", "\n", "* __`Dockerfile`__ describes how to build your Docker container image. More details are provided below.\n", "* __`build_and_push.sh`__ is a script that uses the Dockerfile to build your container images and then pushes it to ECR. We invoke the commands directly later in this notebook, but you can just copy and run the script for your own algorithms.\n", "* __`cifar10`__ is the directory which contains the files that are installed in the container.\n", "\n", "In this simple application, we install only five files in the container. You may only need that many, but if you have many supporting routines, you may wish to install more. These five files show the standard structure of our Python containers, although you are free to choose a different toolset and therefore could have a different layout. If you're writing in a different programming language, you will have a different layout depending on the frameworks and tools you choose.\n", "\n", "The files that we put in the container are:\n", "\n", "* __`cifar10.py`__ is the program that implements our training algorithm.\n", "* __`nginx.conf`__ is the configuration file for the nginx front-end. Generally, you should be able to take this file as-is.\n", "* __`serve`__ is the program started when the container is started for hosting. It simply launches nginx and loads your exported model with TensorFlow Serving.\n", "* __`train`__ is the program that is invoked when the container is run for training. Our implementation of this script invokes cifar10.py with our our hyperparameter values retrieved from /opt/ml/input/config/hyperparameters.json. The goal for doing this is to avoid having to modify our training algorithm program.\n", "\n", "In summary, the two files you probably want to change for your application are `train` and `serve`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The Dockerfile\n", "\n", "The Dockerfile describes the image that we want to build. You can think of it as describing the complete operating system installation of the system that you want to run. A Docker container running is quite a bit lighter than a full operating system, however, because it takes advantage of Linux on the host machine for the basic operations. \n", "\n", "For the Python science stack, we start from an official TensorFlow docker image and run the normal tools to install TensorFlow Serving. Then we add the code that implements our specific algorithm to the container and set up the right environment for it to run under.\n", "\n", "Let's look at the Dockerfile for this example." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "# Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.\r\n", "#\r\n", "# Licensed under the Apache License, Version 2.0 (the \"License\"). You\r\n", "# may not use this file except in compliance with the License. A copy of\r\n", "# the License is located at\r\n", "#\r\n", "# http://aws.amazon.com/apache2.0/\r\n", "#\r\n", "# or in the \"license\" file accompanying this file. This file is\r\n", "# distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF\r\n", "# ANY KIND, either express or implied. See the License for the specific\r\n", "# language governing permissions and limitations under the License.\r\n", "\r\n", "# For more information on creating a Dockerfile\r\n", "# https://docs.docker.com/compose/gettingstarted/#step-2-create-a-dockerfile\r\n", "FROM nvcr.io/nvidia/tensorflow:19.11-tf2-py3\r\n", "\r\n", "RUN apt-get update && apt-get install -y --no-install-recommends nginx curl\r\n", "\r\n", "# Download TensorFlow Serving\r\n", "# https://www.tensorflow.org/serving/setup#installing_the_modelserver\r\n", "RUN echo \"deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\" | tee /etc/apt/sources.list.d/tensorflow-serving.list\r\n", "RUN curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -\r\n", "RUN apt-get update && apt-get install tensorflow-model-server\r\n", "\r\n", "ENV PATH=\"/opt/ml/code:${PATH}\"\r\n", "\r\n", "# /opt/ml and all subdirectories are utilized by SageMaker, we use the /code subdirectory to store our user code.\r\n", "COPY /cifar10 /opt/ml/code\r\n", "WORKDIR /opt/ml/code" ] } ], "source": [ "!cat container/Dockerfile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Building and registering the container\n", "\n", "The following shell code shows how to build the container image using `docker build` and push the container image to ECR using `docker push`. This code is also available as the shell script `container/build-and-push.sh`, which you can run as `build-and-push.sh sagemaker-tf-cifar10-example` to build the image `sagemaker-tf-cifar10-example`. \n", "\n", "This code looks for an ECR repository in the account you're using and the current default region (if you're using a SageMaker notebook instance, this is the region where the notebook instance was created). If the repository doesn't exist, the script will create it." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Login Succeeded\n", "Sending build context to Docker daemon 25.09kB\r", "\r\n", "Step 1/8 : FROM nvcr.io/nvidia/tensorflow:19.11-tf2-py3\n", " ---> 445689316277\n", "Step 2/8 : RUN apt-get update && apt-get install -y --no-install-recommends nginx curl\n", " ---> Using cache\n", " ---> 60a605d8c321\n", "Step 3/8 : RUN echo \"deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal\" | tee /etc/apt/sources.list.d/tensorflow-serving.list\n", " ---> Using cache\n", " ---> ac7710ecc40d\n", "Step 4/8 : RUN curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -\n", " ---> Using cache\n", " ---> fde204d04f33\n", "Step 5/8 : RUN apt-get update && apt-get install tensorflow-model-server\n", " ---> Using cache\n", " ---> 7594bd1700b6\n", "Step 6/8 : ENV PATH=\"/opt/ml/code:${PATH}\"\n", " ---> Using cache\n", " ---> 55c47f6e6d6a\n", "Step 7/8 : COPY /cifar10 /opt/ml/code\n", " ---> cf94ab42fdd6\n", "Step 8/8 : WORKDIR /opt/ml/code\n", " ---> Running in dd08828bf2f8\n", "Removing intermediate container dd08828bf2f8\n", " ---> 6d07088d5bde\n", "Successfully built 6d07088d5bde\n", "Successfully tagged v1-sagemaker-ngc-tf21911-example:latest\n", "The push refers to repository [497456752804.dkr.ecr.us-east-1.amazonaws.com/v1-sagemaker-ngc-tf21911-example]\n", "f85050b18833: Preparing\n", "912ae600de2c: Preparing\n", "73fdbda65eaa: Preparing\n", "d1d4e61c6418: Preparing\n", "f6d872ccc456: Preparing\n", "9dd3122bbb29: Preparing\n", "b0d61cbbf82b: Preparing\n", "b69007ff78b4: Preparing\n", "4da47c3967bd: Preparing\n", "0841a0f66fa2: Preparing\n", "1f8ffba58b5b: Preparing\n", "6237f614c34b: Preparing\n", "9dd3122bbb29: Waiting\n", "39a5ac1ec676: Preparing\n", "b95ba0ce0d05: Preparing\n", "b0d61cbbf82b: Waiting\n", "964ec2207535: Preparing\n", "5316bb670a64: Preparing\n", "b69007ff78b4: Waiting\n", "7ff24a46c319: Preparing\n", "0841a0f66fa2: Waiting\n", "ffe2bf41fa2b: Preparing\n", "4da47c3967bd: Waiting\n", "4ebd4a62fa18: Preparing\n", "aaca97dc6b81: Preparing\n", "6237f614c34b: Waiting\n", "a51305765e40: Preparing\n", "39a5ac1ec676: Waiting\n", "5dda9a74c409: Preparing\n", "2e24bb66a835: Preparing\n", "1f8ffba58b5b: Waiting\n", "964ec2207535: Waiting\n", "5316bb670a64: Waiting\n", "ffe2bf41fa2b: Waiting\n", "459970c695fe: Preparing\n", "bc3622fcfac4: Preparing\n", "062f8292fa36: Preparing\n", "295cd596d59e: Preparing\n", "2e24bb66a835: Waiting\n", "459970c695fe: Waiting\n", "a51305765e40: Waiting\n", "062f8292fa36: Waiting\n", "5dda9a74c409: Waiting\n", "0e2d27f496bf: Preparing\n", "9c8e65977edb: Preparing\n", "bc3622fcfac4: Waiting\n", "0e2d27f496bf: Waiting\n", "be79ee04c7cb: Preparing\n", "295cd596d59e: Waiting\n", "7455d5e9669c: Preparing\n", "9c8e65977edb: Waiting\n", "e7eaf04b0a7c: Preparing\n", "6e87e486530a: Preparing\n", "b1d3bceeb07f: Preparing\n", "07d355b6a56c: Preparing\n", "a0bebe136df7: Preparing\n", "111a107b0d73: Preparing\n", "e66c6ff180c5: Preparing\n", "f71fbaa8d7ac: Preparing\n", "3f04b5de30fe: Preparing\n", "b716a524126b: Preparing\n", "3510e2a8bb18: Preparing\n", "e7eaf04b0a7c: Waiting\n", "cfa9a81e4294: Preparing\n", "e66c6ff180c5: Waiting\n", "6e87e486530a: Waiting\n", "d7cbcee43962: Preparing\n", "a0bebe136df7: Waiting\n", "c37abfc1104a: Preparing\n", "cfa9a81e4294: Waiting\n", "b716a524126b: Waiting\n", "b1d3bceeb07f: Waiting\n", "07d355b6a56c: Waiting\n", "d7cbcee43962: Waiting\n", "7455d5e9669c: Waiting\n", "4a461b14e46a: Preparing\n", "2fd758fbadbe: Preparing\n", "221e6eae7d02: Preparing\n", "29376ed98ab6: Preparing\n", "111a107b0d73: Waiting\n", "221e6eae7d02: Waiting\n", "c37abfc1104a: Waiting\n", "9d95fde9a949: Preparing\n", "58bdc6c643be: Preparing\n", "565c2c6c1ca7: Preparing\n", "adedafa15dc0: Preparing\n", "508c51636b24: Preparing\n", "da726e533907: Preparing\n", "ed4756d335c4: Preparing\n", "3c9a751db107: Preparing\n", "beb23c3afc15: Preparing\n", "5e264a05c849: Preparing\n", "2106c85f8c72: Preparing\n", "de6c5dc71cb4: Preparing\n", "ae4e367d4ed5: Preparing\n", "adef6c78ff23: Preparing\n", "adedafa15dc0: Waiting\n", "2106c85f8c72: Waiting\n", "beb23c3afc15: Waiting\n", "ae4e367d4ed5: Waiting\n", "de6c5dc71cb4: Waiting\n", "58bdc6c643be: Waiting\n", "565c2c6c1ca7: Waiting\n", "ed4756d335c4: Waiting\n", "29376ed98ab6: Waiting\n", "f8f61fdf7ce4: Preparing\n", "f73c85960e15: Preparing\n", "508c51636b24: Waiting\n", "3c9a751db107: Waiting\n", "da726e533907: Waiting\n", "9d95fde9a949: Waiting\n", "e35c1be53e84: Preparing\n", "e0b3afb09dc3: Preparing\n", "6c01b5a53aac: Preparing\n", "2c6ac8e5063e: Preparing\n", "f73c85960e15: Waiting\n", "e35c1be53e84: Waiting\n", "cc967c529ced: Preparing\n", "cc967c529ced: Waiting\n", "912ae600de2c: Layer already exists\n", "73fdbda65eaa: Layer already exists\n", "d1d4e61c6418: Layer already exists\n", "f6d872ccc456: Layer already exists\n", "b0d61cbbf82b: Layer already exists\n", "9dd3122bbb29: Layer already exists\n", "b69007ff78b4: Layer already exists\n", "4da47c3967bd: Layer already exists\n", "0841a0f66fa2: Layer already exists\n", "6237f614c34b: Layer already exists\n", "1f8ffba58b5b: Layer already exists\n", "39a5ac1ec676: Layer already exists\n", "964ec2207535: Layer already exists\n", "b95ba0ce0d05: Layer already exists\n", "5316bb670a64: Layer already exists\n", "7ff24a46c319: Layer already exists\n", "ffe2bf41fa2b: Layer already exists\n", "4ebd4a62fa18: Layer already exists\n", "aaca97dc6b81: Layer already exists\n", "a51305765e40: Layer already exists\n", "5dda9a74c409: Layer already exists\n", "459970c695fe: Layer already exists\n", "2e24bb66a835: Layer already exists\n", "bc3622fcfac4: Layer already exists\n", "062f8292fa36: Layer already exists\n", "295cd596d59e: Layer already exists\n", "0e2d27f496bf: Layer already exists\n", "9c8e65977edb: Layer already exists\n", "be79ee04c7cb: Layer already exists\n", "7455d5e9669c: Layer already exists\n", "e7eaf04b0a7c: Layer already exists\n", "b1d3bceeb07f: Layer already exists\n", "6e87e486530a: Layer already exists\n", "07d355b6a56c: Layer already exists\n", "a0bebe136df7: Layer already exists\n", "111a107b0d73: Layer already exists\n", "e66c6ff180c5: Layer already exists\n", "f71fbaa8d7ac: Layer already exists\n", "3f04b5de30fe: Layer already exists\n", "b716a524126b: Layer already exists\n", "3510e2a8bb18: Layer already exists\n", "cfa9a81e4294: Layer already exists\n", "f85050b18833: Pushed\n", "d7cbcee43962: Layer already exists\n", "4a461b14e46a: Layer already exists\n", "c37abfc1104a: Layer already exists\n", "2fd758fbadbe: Layer already exists\n", "29376ed98ab6: Layer already exists\n", "9d95fde9a949: Layer already exists\n", "221e6eae7d02: Layer already exists\n", "58bdc6c643be: Layer already exists\n", "adedafa15dc0: Layer already exists\n", "565c2c6c1ca7: Layer already exists\n", "508c51636b24: Layer already exists\n", "da726e533907: Layer already exists\n", "ed4756d335c4: Layer already exists\n", "3c9a751db107: Layer already exists\n", "beb23c3afc15: Layer already exists\n", "5e264a05c849: Layer already exists\n", "2106c85f8c72: Layer already exists\n", "de6c5dc71cb4: Layer already exists\n", "ae4e367d4ed5: Layer already exists\n", "f8f61fdf7ce4: Layer already exists\n", "f73c85960e15: Layer already exists\n", "adef6c78ff23: Layer already exists\n", "e35c1be53e84: Layer already exists\n", "6c01b5a53aac: Layer already exists\n", "e0b3afb09dc3: Layer already exists\n", "2c6ac8e5063e: Layer already exists\n", "cc967c529ced: Layer already exists\n", "latest: digest: sha256:4207b797cb63641f95f7dc42e1414678c8c949f83838e460c79f8cee92ffa1d7 size: 14963\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING! Using --password via the CLI is insecure. Use --password-stdin.\n", "WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.\n", "Configure a credential helper to remove this warning. See\n", "https://docs.docker.com/engine/reference/commandline/login/#credentials-store\n", "\n" ] } ], "source": [ "%%sh\n", "\n", "# The name of our algorithm\n", "algorithm_name=v1-sagemaker-ngc-tf21911-example\n", "\n", "cd container\n", "\n", "chmod +x cifar10/train\n", "chmod +x cifar10/serve\n", "\n", "account=$(aws sts get-caller-identity --query Account --output text)\n", "\n", "# Get the region defined in the current configuration (default to us-west-2 if none defined)\n", "region=$(aws configure get region)\n", "region=${region:-us-west-2}\n", "\n", "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n", "\n", "# If the repository doesn't exist in ECR, create it.\n", "\n", "aws ecr describe-repositories --repository-names \"${algorithm_name}\" > /dev/null 2>&1\n", "\n", "if [ $? -ne 0 ]\n", "then\n", " aws ecr create-repository --repository-name \"${algorithm_name}\" > /dev/null\n", "fi\n", "\n", "# Get the login command from ECR and execute it directly\n", "$(aws ecr get-login --region ${region} --no-include-email)\n", "\n", "# Build the docker image locally with the image name and then push it to ECR\n", "# with the full name.\n", "\n", "docker build -t ${algorithm_name} .\n", "docker tag ${algorithm_name} ${fullname}\n", "\n", "docker push ${fullname}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Testing your algorithm on your local machine\n", "\n", "When you're packaging you first algorithm to use with Amazon SageMaker, you probably want to test it yourself to make sure it's working correctly. We use the [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk) to test both locally and on SageMaker. For more examples with the SageMaker Python SDK, see [Amazon SageMaker Examples](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/sagemaker-python-sdk). In order to test our algorithm, we need our dataset." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download the CIFAR-10 dataset\n", "Our training algorithm is expecting our training data to be in the file format of [TFRecords](https://www.tensorflow.org/guide/datasets), which is a simple record-oriented binary format that many TensorFlow applications use for training data.\n", "Below is a Python script adapted from the [official TensorFlow CIFAR-10 example](https://github.com/tensorflow/models/tree/master/tutorials/image/cifar10_estimator), which downloads the CIFAR-10 dataset and converts them into TFRecords." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/__init__.py:1467: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.\n", "\n", "Download from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz and extract.\n", "FloatProgress(value=0.0)\n", ">> Downloading cifar-10-python.tar.gz \n", "Successfully downloaded cifar-10-python.tar.gz 170498071 bytes.\n", "Generating /tmp/cifar-10-data/train.tfrecords\n", "WARNING:tensorflow:From utils/generate_cifar10_tfrecords.py:99: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n", "\n", "WARNING:tensorflow:From utils/generate_cifar10_tfrecords.py:88: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n", "\n", "Generating /tmp/cifar-10-data/validation.tfrecords\n", "Generating /tmp/cifar-10-data/eval.tfrecords\n", "Removing original files.\n", "Done!\n" ] } ], "source": [ "! python utils/generate_cifar10_tfrecords.py --data-dir=/tmp/cifar-10-data" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "eval.tfrecords\ttrain.tfrecords validation.tfrecords\r\n" ] } ], "source": [ "# There should be three tfrecords. (eval, train, validation)\n", "! ls /tmp/cifar-10-data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## SageMaker Python SDK Local Training\n", "To represent our training, we use the Estimator class, which needs to be configured in five steps. \n", "1. IAM role - our AWS execution role\n", "2. train_instance_count - number of instances to use for training.\n", "3. train_instance_type - type of instance to use for training. For training locally, we specify `local`.\n", "4. image_name - our custom TensorFlow Docker image we created.\n", "5. hyperparameters - hyperparameters we want to pass.\n", "\n", "Let's start with setting up our IAM role. We make use of a helper function within the Python SDK. This function throw an exception if run outside of a SageMaker notebook instance, as it gets metadata from the notebook instance. If running outside, you must provide an IAM role with proper access stated above in [Permissions](#Permissions)." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "from sagemaker import get_execution_role\n", "\n", "role = get_execution_role()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit, Deploy, Predict\n", "\n", "Now that the rest of our estimator is configured, we can call `fit()` with the path to our local CIFAR10 dataset prefixed with `file://`. This invokes our TensorFlow container with 'train' and passes in our hyperparameters and other metadata as json files in /opt/ml/input/config within the container.\n", "\n", "After our training has succeeded, our training algorithm outputs our trained model within the /opt/ml/model directory, which is used to handle predictions.\n", "\n", "We can then call `deploy()` with an instance_count and instance_type, which is 1 and `local`. This invokes our Tensorflow container with 'serve', which setups our container to handle prediction requests through TensorFlow Serving. What is returned is a predictor, which is used to make inferences against our trained model.\n", "\n", "After our prediction, we can delete our endpoint.\n", "\n", "We recommend testing and training your training algorithm locally first, as it provides quicker iterations and better debuggability." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "The user has root access.\n", "nvidia-docker2 already installed. We are good to go!\n", "SageMaker instance route table setup is ok. We are good to go.\n", "SageMaker instance routing for Docker is ok. We are good to go!\n" ] } ], "source": [ "# Lets set up our SageMaker notebook instance for local mode.\n", "!/bin/bash ./utils/setup.sh" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Creating tmpvaotpc2p_algo-1-p40ve_1 ... \n", "\u001b[1BAttaching to tmpvaotpc2p_algo-1-p40ve_12mdone\u001b[0m\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m ================\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m == TensorFlow ==\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m ================\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m NVIDIA Release 19.11-tf2 (build 8776033)\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m TensorFlow Version 2.0.0\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m Container image Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m Copyright 2017-2019 The TensorFlow Authors. All rights reserved.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m NVIDIA modifications are covered by the license terms that apply to the underlying project or file.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m Use 'nvidia-docker run' to start this container; see\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m NOTE: MOFED driver for multi-node communication was not detected.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m Multi-node communication performance may be reduced.\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m insufficient for TensorFlow. NVIDIA recommends the use of the following flags:\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 ...\n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m \n", "\u001b[36malgo-1-p40ve_1 |\u001b[0m Training complete.\n", "\u001b[36mtmpvaotpc2p_algo-1-p40ve_1 exited with code 0\n", "\u001b[0mAborting on container exit...\n", "===== Job Complete =====\n", "!" ] } ], "source": [ "from sagemaker.estimator import Estimator\n", "\n", "hyperparameters = {'train-steps': 100}\n", "\n", "instance_type = 'local'\n", "\n", "estimator = Estimator(role=role,\n", " train_instance_count=1,\n", " train_instance_type=instance_type,\n", " image_name='v1-sagemaker-ngc-tf21911-example',\n", " hyperparameters=hyperparameters)\n", "\n", "estimator.fit('file:///tmp/cifar-10-data')\n", "\n", "predictor = estimator.deploy(1, instance_type)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Making predictions using Python SDK\n", "\n", "To make predictions, we use an image that is converted using OpenCV into a json format to send as an inference request. We need to install OpenCV to deserialize the image that is used to make predictions.\n", "\n", "The JSON reponse will be the probabilities of the image belonging to one of the 10 classes along with the most likely class the picture belongs to. The classes can be referenced from the [CIFAR-10 website](https://www.cs.toronto.edu/~kriz/cifar.html). Since we didn't train the model for that long, we aren't expecting very accurate results." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: opencv-python in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (3.4.2.17)\n", "Requirement already satisfied: numpy>=1.11.3 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from opencv-python) (1.16.4)\n", "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.1 is available.\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n" ] } ], "source": [ "! pip install opencv-python" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'tfeager': False, 'output': ['not enough image data']}" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import cv2\n", "import numpy\n", "\n", "from sagemaker.predictor import json_serializer, json_deserializer\n", "\n", "image = cv2.imread(\"data/cat.png\", 1)\n", "\n", "# resize, as our model is expecting images in 32x32.\n", "image = cv2.resize(image, (32, 32))\n", "\n", "data = {'instances': numpy.asarray(image).astype(float).tolist()}\n", "\n", "# The request and response format is JSON for TensorFlow Serving.\n", "# For more information: https://www.tensorflow.org/serving/api_rest#predict_api\n", "predictor.accept = 'application/json'\n", "predictor.content_type = 'application/json'\n", "\n", "predictor.serializer = json_serializer\n", "predictor.deserializer = json_deserializer\n", "\n", "# For more information on the predictor class.\n", "# https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/predictor.py\n", "predictor.predict(data)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Gracefully stopping... (press Ctrl+C again to force)\n" ] } ], "source": [ "predictor.delete_endpoint()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Part 2: Training and Hosting your Algorithm in Amazon SageMaker\n", "Once you have your container packaged, you can use it to train and serve models. Let's do that with the algorithm we made above.\n", "\n", "## Set up the environment\n", "Here we specify the bucket to use and the role that is used for working with SageMaker." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# S3 prefix\n", "prefix = 'DEMO-tensorflow-cifar10'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create the session\n", "\n", "The session remembers our connection parameters to SageMaker. We use it to perform all of our SageMaker operations." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "import sagemaker as sage\n", "\n", "sess = sage.Session()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Upload the data for training\n", "\n", "We will use the tools provided by the SageMaker Python SDK to upload the data to a default bucket." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "WORK_DIRECTORY = '/tmp/cifar-10-data'\n", "\n", "data_location = sess.upload_data(WORK_DIRECTORY, key_prefix=prefix)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training on SageMaker\n", "Training a model on SageMaker with the Python SDK is done in a way that is similar to the way we trained it locally. This is done by changing our train_instance_type from `local` to one of our [supported EC2 instance types](https://aws.amazon.com/sagemaker/pricing/instance-types/).\n", "\n", "In addition, we must now specify the ECR image URL, which we just pushed above.\n", "\n", "Finally, our local training dataset has to be in Amazon S3 and the S3 URL to our dataset is passed into the `fit()` call.\n", "\n", "Let's first fetch our ECR image url that corresponds to the image we just built and pushed." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "497456752804.dkr.ecr.us-east-1.amazonaws.com/v1-sagemaker-ngc-tf21911-example:latest\n" ] } ], "source": [ "import boto3\n", "\n", "client = boto3.client('sts')\n", "account = client.get_caller_identity()['Account']\n", "\n", "my_session = boto3.session.Session()\n", "region = my_session.region_name\n", "\n", "algorithm_name = 'v1-sagemaker-ngc-tf21911-example'\n", "\n", "ecr_image = '{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(account, region, algorithm_name)\n", "\n", "print(ecr_image)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2020-05-08 20:57:54 Starting - Starting the training job...\n", "2020-05-08 20:57:56 Starting - Launching requested ML instances......\n", "2020-05-08 20:59:04 Starting - Preparing the instances for training......\n", "2020-05-08 21:00:12 Downloading - Downloading input data...\n", "2020-05-08 21:00:43 Training - Downloading the training image...............\n", "2020-05-08 21:03:25 Training - Training image download completed. Training in progress..\u001b[34m================\u001b[0m\n", "\u001b[34m== TensorFlow ==\u001b[0m\n", "\u001b[34m================\n", "\u001b[0m\n", "\u001b[34mNVIDIA Release 19.11-tf2 (build 8776033)\u001b[0m\n", "\u001b[34mTensorFlow Version 2.0.0\n", "\u001b[0m\n", "\u001b[34mContainer image Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.\u001b[0m\n", "\u001b[34mCopyright 2017-2019 The TensorFlow Authors. All rights reserved.\n", "\u001b[0m\n", "\u001b[34mVarious files include modifications (c) NVIDIA CORPORATION. All rights reserved.\u001b[0m\n", "\u001b[34mNVIDIA modifications are covered by the license terms that apply to the underlying project or file.\u001b[0m\n", "\u001b[34mNOTE: MOFED driver for multi-node communication was not detected.\n", " Multi-node communication performance may be reduced.\n", "\u001b[0m\n", "\u001b[34mTraining complete.\u001b[0m\n", "\n", "2020-05-08 21:05:06 Uploading - Uploading generated training model\n", "2020-05-08 21:06:13 Completed - Training job completed\n", "Training seconds: 361\n", "Billable seconds: 361\n", "-------------------!" ] } ], "source": [ "from sagemaker.estimator import Estimator\n", "\n", "hyperparameters = {'train-steps': 1}\n", "\n", "instance_type = 'ml.p3.8xlarge'\n", "\n", "estimator = Estimator(role=role,\n", " train_instance_count=1,\n", " train_instance_type=instance_type,\n", " image_name=ecr_image,\n", " hyperparameters=hyperparameters)\n", "\n", "estimator.fit(data_location)\n", "\n", "predictor = estimator.deploy(1, instance_type)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "import cv2\n", "import numpy\n", "from sagemaker.predictor import json_serializer, json_deserializer" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'predictions': [{'probabilities': [1.0,\n", " 2.08015345e-36,\n", " 0.0,\n", " 3.33216129e-37,\n", " 0.0,\n", " 1.76619644e-38,\n", " 0.0,\n", " 5.2339639e-33,\n", " 1.59953973e-36,\n", " 0.0],\n", " 'classes': 0}]}" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "image = cv2.imread(\"data/cat.png\", 1)\n", "\n", "# resize, as our model is expecting images in 32x32.\n", "image = cv2.resize(image, (32, 32))\n", "\n", "data = {'instances': numpy.asarray(image).astype(float).tolist()}\n", "\n", "predictor.accept = 'application/json'\n", "predictor.content_type = 'application/json'\n", "\n", "predictor.serializer = json_serializer\n", "predictor.deserializer = json_deserializer\n", "\n", "predictor.predict(data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optional cleanup\n", "When you're done with the endpoint, you should clean it up.\n", "\n", "All of the training jobs, models and endpoints we created can be viewed through the SageMaker console of your AWS account." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "predictor.delete_endpoint()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Reference\n", "- [How Amazon SageMaker interacts with your Docker container for training](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html)\n", "- [How Amazon SageMaker interacts with your Docker container for inference](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html)\n", "- [CIFAR-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html)\n", "- [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk)\n", "- [Dockerfile](https://docs.docker.com/engine/reference/builder/)\n", "- [scikit-bring-your-own](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "conda_tensorflow_p36", "language": "python", "name": "conda_tensorflow_p36" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 2 }