{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Basic Custom Training Container
"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook demonstrates how to build and use a basic custom Docker container for training with Amazon SageMaker. Reference documentation is available at https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We start by defining some variables like the current execution role, the ECR repository that we are going to use for pushing the custom Docker container and a default Amazon S3 bucket to be used by Amazon SageMaker."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"import sagemaker\n",
"from sagemaker import get_execution_role\n",
"\n",
"ecr_namespace = \"sagemaker-training-containers/\"\n",
"prefix = \"basic-training-container\"\n",
"\n",
"ecr_repository_name = ecr_namespace + prefix\n",
"role = get_execution_role()\n",
"account_id = role.split(\":\")[4]\n",
"region = boto3.Session().region_name\n",
"sagemaker_session = sagemaker.session.Session()\n",
"bucket = sagemaker_session.default_bucket()\n",
"\n",
"print(account_id)\n",
"print(region)\n",
"print(role)\n",
"print(bucket)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's take a look at the Dockerfile which defines the statements for building our custom SageMaker training container:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pygmentize ../docker/Dockerfile"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At high-level the Dockerfile specifies the following operations for building this container:\n",
"\n",
" - Start from Ubuntu 16.04
\n",
" - Define some variables to be used at build time to install Python 3
\n",
" - Some handful libraries are installed with apt-get
\n",
" - We then install Python 3 and create a symbolic link
\n",
" - We install some Python libraries like numpy, pandas, ScikitLearn, etc.
\n",
" - We set e few environment variables, including PYTHONUNBUFFERED which is used to avoid buffering Python standard output (useful for logging)
\n",
" - Finally, we copy all contents in code/ (which is where our training code is) to the WORKDIR and define the ENTRYPOINT
\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Build and push the container
\n",
"We are now ready to build this container and push it to Amazon ECR. This task is executed using a shell script stored in the ../script/ folder. Let's take a look at this script and then execute it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pygmentize ../scripts/build_and_push.sh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"--------------------------------------------------------------------------------------------------------------------
\n",
"\n",
"The script builds the Docker container, then creates the repository if it does not exist, and finally pushes the container to the ECR repository. The build task requires a few minutes to be executed the first time, then Docker caches build outputs to be reused for the subsequent build operations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"! ../scripts/build_and_push.sh $account_id $region $ecr_repository_name"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Training with Amazon SageMaker
\n",
"\n",
"Once we have correctly pushed our container to Amazon ECR, we are ready to start training with Amazon SageMaker, which requires the ECR path to the Docker container used for training as parameter for starting a training job."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"container_image_uri = \"{0}.dkr.ecr.{1}.amazonaws.com/{2}:latest\".format(\n",
" account_id, region, ecr_repository_name\n",
")\n",
"print(container_image_uri)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given the purpose of this example is explaining how to build custom containers, we are not going to train a real model. The script that will be executed does not define a specific training logic; it just outputs the configurations injected by SageMaker and implements a dummy training loop. Training data is also dummy. Let's analyze the code first:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! pygmentize ../docker/code/main.py"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We upload some dummy data to Amazon S3, in order to define our S3-based training channels."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"! echo \"val1, val2, val3\" > dummy.csv\n",
"print(sagemaker_session.upload_data(\"dummy.csv\", bucket, prefix + \"/train\"))\n",
"print(sagemaker_session.upload_data(\"dummy.csv\", bucket, prefix + \"/val\"))\n",
"! rm dummy.csv"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we can execute the training job by calling the fit() method of the generic Estimator object defined in the Amazon SageMaker Python SDK (https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/estimator.py). This corresponds to calling the CreateTrainingJob() API (https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sagemaker\n",
"\n",
"est = sagemaker.estimator.Estimator(\n",
" container_image_uri,\n",
" role,\n",
" train_instance_count=1,\n",
" train_instance_type=\"local\", # use local mode\n",
" # train_instance_type='ml.m5.xlarge',\n",
" base_job_name=prefix,\n",
")\n",
"\n",
"est.set_hyperparameters(hp1=\"value1\", hp2=300, hp3=0.001)\n",
"\n",
"train_config = sagemaker.session.s3_input(\n",
" \"s3://{0}/{1}/train/\".format(bucket, prefix), content_type=\"text/csv\"\n",
")\n",
"val_config = sagemaker.session.s3_input(\n",
" \"s3://{0}/{1}/val/\".format(bucket, prefix), content_type=\"text/csv\"\n",
")\n",
"\n",
"est.fit({\"train\": train_config, \"validation\": val_config})"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "conda_python3",
"language": "python",
"name": "conda_python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}