{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

Script-mode Custom Training Container (2)

" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-2/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This notebook demonstrates how to build and use a custom Docker container for training with Amazon SageMaker that leverages on the Script Mode execution that is implemented by the sagemaker-training-toolkit library. Reference documentation is available at https://github.com/aws/sagemaker-training-toolkit.\n", "\n", "The difference from the first example is that we are not copying the training code during the Docker build process, and we are loading them dynamically from Amazon S3 (this feature is implemented through the sagemaker-training-toolkit)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We start by defining some variables like the current execution role, the ECR repository that we are going to use for pushing the custom Docker container and a default Amazon S3 bucket to be used by Amazon SageMaker." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "import sagemaker\n", "from sagemaker import get_execution_role\n", "\n", "ecr_namespace = \"sagemaker-training-containers/\"\n", "prefix = \"script-mode-container-2\"\n", "\n", "ecr_repository_name = ecr_namespace + prefix\n", "role = get_execution_role()\n", "account_id = role.split(\":\")[4]\n", "region = boto3.Session().region_name\n", "sagemaker_session = sagemaker.session.Session()\n", "bucket = sagemaker_session.default_bucket()\n", "\n", "print(account_id)\n", "print(region)\n", "print(role)\n", "print(bucket)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at the Dockerfile which defines the statements for building our script-mode custom training container:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[37m# Part of the implementation of this container is based on the Amazon SageMaker Apache MXNet container.\u001b[39;49;00m\n", "\u001b[37m# https://github.com/aws/sagemaker-mxnet-container\u001b[39;49;00m\n", "\n", "\u001b[34mFROM\u001b[39;49;00m\u001b[33m ubuntu:16.04\u001b[39;49;00m\n", "\n", "LABEL \u001b[31mmaintainer\u001b[39;49;00m=\u001b[33m\"Giuseppe A. Porcelli\"\u001b[39;49;00m\n", "\n", "\u001b[37m# Defining some variables used at build time to install Python3\u001b[39;49;00m\n", "ARG \u001b[31mPYTHON\u001b[39;49;00m=python3\n", "ARG \u001b[31mPYTHON_PIP\u001b[39;49;00m=python3-pip\n", "ARG \u001b[31mPIP\u001b[39;49;00m=pip3\n", "ARG \u001b[31mPYTHON_VERSION\u001b[39;49;00m=\u001b[34m3\u001b[39;49;00m.6.6\n", "\n", "\u001b[37m# Install some handful libraries like curl, wget, git, build-essential, zlib\u001b[39;49;00m\n", "\u001b[34mRUN\u001b[39;49;00m apt-get update && apt-get install -y --no-install-recommends software-properties-common && \u001b[33m\\\u001b[39;49;00m\n", " add-apt-repository ppa:deadsnakes/ppa -y && \u001b[33m\\\u001b[39;49;00m\n", " apt-get update && apt-get install -y --no-install-recommends \u001b[33m\\\u001b[39;49;00m\n", " build-essential \u001b[33m\\\u001b[39;49;00m\n", " ca-certificates \u001b[33m\\\u001b[39;49;00m\n", " curl \u001b[33m\\\u001b[39;49;00m\n", " wget \u001b[33m\\\u001b[39;49;00m\n", " git \u001b[33m\\\u001b[39;49;00m\n", " libopencv-dev \u001b[33m\\\u001b[39;49;00m\n", " openssh-client \u001b[33m\\\u001b[39;49;00m\n", " openssh-server \u001b[33m\\\u001b[39;49;00m\n", " vim \u001b[33m\\\u001b[39;49;00m\n", " zlib1g-dev && \u001b[33m\\\u001b[39;49;00m\n", " rm -rf /var/lib/apt/lists/*\n", "\n", "\u001b[37m# Installing Python3\u001b[39;49;00m\n", "\u001b[34mRUN\u001b[39;49;00m wget https://www.python.org/ftp/python/\u001b[31m$PYTHON_VERSION\u001b[39;49;00m/Python-\u001b[31m$PYTHON_VERSION\u001b[39;49;00m.tgz && \u001b[33m\\\u001b[39;49;00m\n", " tar -xvf Python-\u001b[31m$PYTHON_VERSION\u001b[39;49;00m.tgz && \u001b[36mcd\u001b[39;49;00m Python-\u001b[31m$PYTHON_VERSION\u001b[39;49;00m && \u001b[33m\\\u001b[39;49;00m\n", " ./configure && make && make install && \u001b[33m\\\u001b[39;49;00m\n", " apt-get update && apt-get install -y --no-install-recommends libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev && \u001b[33m\\\u001b[39;49;00m\n", " make && make install && rm -rf ../Python-\u001b[31m$PYTHON_VERSION\u001b[39;49;00m* && \u001b[33m\\\u001b[39;49;00m\n", " ln -s /usr/local/bin/pip3 /usr/bin/pip\n", "\n", "\u001b[37m# Upgrading pip and creating symbolic link for python3\u001b[39;49;00m\n", "\u001b[34mRUN\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mPIP\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m --no-cache-dir install --upgrade pip\n", "\u001b[34mRUN\u001b[39;49;00m ln -s \u001b[34m$(\u001b[39;49;00mwhich \u001b[33m${\u001b[39;49;00m\u001b[31mPYTHON\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[34m)\u001b[39;49;00m /usr/local/bin/python\n", "\n", "\u001b[34mWORKDIR\u001b[39;49;00m\u001b[33m /\u001b[39;49;00m\n", "\n", "\u001b[37m# Installing numpy, pandas, scikit-learn, scipy\u001b[39;49;00m\n", "\u001b[34mRUN\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mPIP\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m install --no-cache --upgrade \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mnumpy\u001b[39;49;00m==\u001b[34m1\u001b[39;49;00m.14.5 \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mpandas\u001b[39;49;00m==\u001b[34m0\u001b[39;49;00m.24.1 \u001b[33m\\\u001b[39;49;00m\n", " scikit-learn==\u001b[34m0\u001b[39;49;00m.20.3 \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mrequests\u001b[39;49;00m==\u001b[34m2\u001b[39;49;00m.21.0 \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mscipy\u001b[39;49;00m==\u001b[34m1\u001b[39;49;00m.2.2\n", "\n", "\u001b[37m# Setting some environment variables.\u001b[39;49;00m\n", "\u001b[34mENV\u001b[39;49;00m\u001b[33m PYTHONDONTWRITEBYTECODE=1 \\\u001b[39;49;00m\n", " \u001b[31mPYTHONUNBUFFERED\u001b[39;49;00m=\u001b[34m1\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mLD_LIBRARY_PATH\u001b[39;49;00m=\u001b[33m\"\u001b[39;49;00m\u001b[33m${\u001b[39;49;00m\u001b[31mLD_LIBRARY_PATH\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m\u001b[33m:/usr/local/lib\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mPYTHONIOENCODING\u001b[39;49;00m=UTF-8 \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mLANG\u001b[39;49;00m=C.UTF-8 \u001b[33m\\\u001b[39;49;00m\n", " \u001b[31mLC_ALL\u001b[39;49;00m=C.UTF-8\n", "\n", "\u001b[34mRUN\u001b[39;49;00m \u001b[33m${\u001b[39;49;00m\u001b[31mPIP\u001b[39;49;00m\u001b[33m}\u001b[39;49;00m install --no-cache --upgrade \u001b[33m\\\u001b[39;49;00m\n", " sagemaker-training\n" ] } ], "source": [ "! pygmentize ../docker/Dockerfile" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At high-level the Dockerfile specifies the following operations for building this container:\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Build and push the container

\n", "We are now ready to build this container and push it to Amazon ECR. This task is executed using a shell script stored in the ../script/ folder. Let's take a look at this script and then execute it." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "! pygmentize ../scripts/build_and_push.sh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

--------------------------------------------------------------------------------------------------------------------

\n", "\n", "The script builds the Docker container, then creates the repository if it does not exist, and finally pushes the container to the ECR repository. The build task requires a few minutes to be executed the first time, then Docker caches build outputs to be reused for the subsequent build operations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture\n", "! ../scripts/build_and_push.sh $account_id $region $ecr_repository_name" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Training with Amazon SageMaker

\n", "\n", "Once we have correctly pushed our container to Amazon ECR, we are ready to start training with Amazon SageMaker, which requires the ECR path to the Docker container used for training as parameter for starting a training job." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "container_image_uri = \"{0}.dkr.ecr.{1}.amazonaws.com/{2}:latest\".format(\n", " account_id, region, ecr_repository_name\n", ")\n", "print(container_image_uri)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Given the purpose of this example is explaining how to build custom script-mode containers, we are not going to train a real model. The script that will be executed does not define a specific training logic; it just outputs the configurations injected by SageMaker and implements a dummy training loop. Training data is also dummy. Let's analyze the script first:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "! pygmentize source_dir/train.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can realize that the training code has been implemented as a standard Python script, that will be invoked by the sagemaker-training-toolkit library passing hyperparameters as arguments. This way of invoking training script is indeed called Script Mode for Amazon SageMaker containers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we upload some dummy data to Amazon S3, in order to define our S3-based training channels." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "! echo \"val1, val2, val3\" > dummy.csv\n", "print(sagemaker_session.upload_data(\"dummy.csv\", bucket, prefix + \"/train\"))\n", "print(sagemaker_session.upload_data(\"dummy.csv\", bucket, prefix + \"/val\"))\n", "! rm dummy.csv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We want to dynamically run user-provided code loading it from Amazon S3, so we need to:\n", "\n", "\n", "Note: these steps are executed automatically by the Amazon SageMaker Python SDK when using framework estimators for MXNet, Tensorflow, etc." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import tarfile\n", "import os\n", "\n", "\n", "def create_tar_file(source_files, target=None):\n", " if target:\n", " filename = target\n", " else:\n", " _, filename = tempfile.mkstemp()\n", "\n", " with tarfile.open(filename, mode=\"w:gz\") as t:\n", " for sf in source_files:\n", " # Add all files from the directory into the root of the directory structure of the tar\n", " t.add(sf, arcname=os.path.basename(sf))\n", " return filename\n", "\n", "\n", "create_tar_file([\"source_dir/train.py\", \"source_dir/utils.py\"], \"sourcedir.tar.gz\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sources = sagemaker_session.upload_data(\"sourcedir.tar.gz\", bucket, prefix + \"/code\")\n", "print(sources)\n", "! rm sourcedir.tar.gz" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When starting the training job, we need to let the sagemaker-training-toolkit library know where the sources are stored in Amazon S3 and what is the module to be invoked. These parameters are specified through the following reserved hyperparameters (these reserved hyperparameters are injected automatically when using framework estimators of the Amazon SageMaker Python SDK):\n", "\n", "\n", "Finally, we can execute the training job by calling the fit() method of the generic Estimator object defined in the Amazon SageMaker Python SDK (https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/estimator.py). This corresponds to calling the CreateTrainingJob() API (https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import sagemaker\n", "import json\n", "\n", "# JSON encode hyperparameters.\n", "def json_encode_hyperparameters(hyperparameters):\n", " return {str(k): json.dumps(v) for (k, v) in hyperparameters.items()}\n", "\n", "\n", "hyperparameters = json_encode_hyperparameters(\n", " {\n", " \"sagemaker_program\": \"train.py\",\n", " \"sagemaker_submit_directory\": sources,\n", " \"hp1\": \"value1\",\n", " \"hp2\": 300,\n", " \"hp3\": 0.001,\n", " }\n", ")\n", "\n", "est = sagemaker.estimator.Estimator(\n", " container_image_uri,\n", " role,\n", " train_instance_count=1,\n", " train_instance_type=\"local\",\n", " base_job_name=prefix,\n", " hyperparameters=hyperparameters,\n", ")\n", "\n", "train_config = sagemaker.session.s3_input(\n", " \"s3://{0}/{1}/train/\".format(bucket, prefix), content_type=\"text/csv\"\n", ")\n", "val_config = sagemaker.session.s3_input(\n", " \"s3://{0}/{1}/val/\".format(bucket, prefix), content_type=\"text/csv\"\n", ")\n", "\n", "est.fit({\"train\": train_config, \"validation\": val_config})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Training with a custom SDK framework estimator

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you have seen, in the previous steps we had to upload our code to Amazon S3 and then inject reserved hyperparameters to execute training. In order to facilitate this task, you can also try defining a custom framework estimator using the Amazon SageMaker Python SDK and run training with that class, which will take care of managing these tasks." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.estimator import Framework\n", "\n", "\n", "class CustomFramework(Framework):\n", " def __init__(\n", " self,\n", " entry_point,\n", " source_dir=None,\n", " hyperparameters=None,\n", " py_version=\"py3\",\n", " framework_version=None,\n", " image_name=None,\n", " distributions=None,\n", " **kwargs,\n", " ):\n", " super(CustomFramework, self).__init__(\n", " entry_point, source_dir, hyperparameters, image_name=image_name, **kwargs\n", " )\n", "\n", " def _configure_distribution(self, distributions):\n", " return\n", "\n", " def create_model(\n", " self,\n", " model_server_workers=None,\n", " role=None,\n", " vpc_config_override=None,\n", " entry_point=None,\n", " source_dir=None,\n", " dependencies=None,\n", " image_name=None,\n", " **kwargs,\n", " ):\n", " return None\n", "\n", "\n", "import sagemaker\n", "\n", "est = CustomFramework(\n", " image_name=container_image_uri,\n", " role=role,\n", " entry_point=\"train.py\",\n", " source_dir=\"source_dir/\",\n", " train_instance_count=1,\n", " train_instance_type=\"local\", # we use local mode\n", " # train_instance_type='ml.m5.xlarge',\n", " base_job_name=prefix,\n", " hyperparameters={\"hp1\": \"value1\", \"hp2\": \"300\", \"hp3\": \"0.001\"},\n", ")\n", "\n", "train_config = sagemaker.session.s3_input(\n", " \"s3://{0}/{1}/train/\".format(bucket, prefix), content_type=\"text/csv\"\n", ")\n", "val_config = sagemaker.session.s3_input(\n", " \"s3://{0}/{1}/val/\".format(bucket, prefix), content_type=\"text/csv\"\n", ")\n", "\n", "est.fit({\"train\": train_config, \"validation\": val_config})" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-east-2/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/us-west-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ca-central-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/sa-east-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-2/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-west-3/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-central-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/eu-north-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-southeast-2/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-northeast-2/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://h75twx4l60.execute-api.us-west-2.amazonaws.com/sagemaker-nb/ap-south-1/advanced_functionality|custom-training-containers|script-mode-container-2|notebook|script-mode-container-2.ipynb)\n" ] } ], "metadata": { "kernelspec": { "display_name": "conda_python3", "language": "python", "name": "conda_python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 4 }