{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Build a Custom Training Container and Debug Training Jobs with Amazon SageMaker Debugger"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Amazon SageMaker Debugger enables you to debug your model through its built-in rules and tools (`smdebug` hook and core features) to store and retrieve output tensors in Amazon Simple Storage Service (S3). \n",
"To run your customized machine learning/deep learning (ML/DL) models, use Amazon Elastic Container Registry (ECR) to build and push your customized training container. \n",
"Use SageMaker Debugger for training jobs run on Amazon EC2 instance and take the benefit of its built-in functionalities.\n",
"\n",
"You can bring your own model customized with state-of-the-art ML/DL frameworks, such as TensorFlow, PyTorch, MXNet, and XGBoost. \n",
"You can also use your Docker base image or AWS Deep Learning Container base images to build a custom training container.\n",
"To run and debug your training script using SageMaker Debugger, you need to register the Debugger hook to the script.\n",
"Using the `smdebug` trial feature, you can retrieve the output tensors and visualize it for analysis.\n",
"\n",
"By monitoring the output tensors, the Debugger rules detect training issues and invoke a `IssueFound` rule job status. \n",
"The rule job status also returns at which step or epoch the training job started having the issues. \n",
"You can send this invoked status to Amazon CloudWatch and AWS Lambda to stop the training job when the Debugger rule triggers the `IssueFound` status.\n",
"\n",
"The workflow is as follows:\n",
"\n",
"- [Step 1: Prepare prerequisites](#step1)\n",
"- [Step 2: Prepare a Dockerfile and register the Debugger hook to you training script](#step2)\n",
"- [Step 3: Create a Docker image, build the Docker training container, and push to Amazon ECR](#step3)\n",
"- [Step 4: Use Amazon SageMaker to set the Debugger hook and rule configuration](#step4)\n",
"- [Step 5: Define a SageMaker Estimator object with Debugger and initiate a training job](#step5)\n",
"- [Step 6: Retrieve output tensors using the smdebug trials class](#step6)\n",
"- [Step 7: Analyze the training job using the smdebug trial methods and rule job status](#step7)\n",
"\n",
"**Important:** You can run this notebook only on SageMaker Notebook instances. You cannot run this in SageMaker Studio. Studio does not support Docker container build."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Prepare prerequisites\n",
"\n",
"### Install the SageMaker Python SDK v2 and the smdebug library\n",
"\n",
"This notebook runs on the latest version of the SageMaker Python SDK and the `smdebug` client library. If you want to use one of the previous version, specify the version number for installation. For example, `pip install sagemaker==x.xx.0`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"!{sys.executable} -m pip install \"sagemaker==1.72.0\" smdebug"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Optional Step] Restart the kernel to apply the update"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note:** If you are using **Jupyter Notebook**, the previous cell automatically installs and updates the libraries. If you are using **JupyterLab**, you have to manually choose the \"Restart Kernel\" under the **Kernel** tab in the top menu bar.\n",
"\n",
"Check the SageMaker Python SDK version by running the following cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sagemaker\n",
"\n",
"sagemaker.__version__"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Prepare a Dockerfile and register the Debugger hook to you training script\n",
"\n",
"You need to put your **Dockerfile** and training script (**tf_keras_resnet_byoc.py** in this case) in the **docker** folder. Specify the location of the training script in the **Dockerfile** script in the line for `COPY` and `ENV`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare a Dockerfile\n",
"\n",
"The following cell prints the **Dockerfile** in the **docker** folder. You must install `sagemaker-training` and `smdebug` libraries to fully access the SageMaker Debugger features."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[34mFROM\u001b[39;49;00m \u001b[33mtensorflow/tensorflow:2.2.0rc2-py3-jupyter\u001b[39;49;00m\r\n",
"\r\n",
"\u001b[37m# Install Amazon SageMaker training toolkit and smdebug libraries\u001b[39;49;00m\r\n",
"\u001b[34mRUN\u001b[39;49;00m pip install sagemaker-training\r\n",
"\u001b[34mRUN\u001b[39;49;00m pip install smdebug\r\n",
"\r\n",
"\u001b[37m# Copies the training code inside the container\u001b[39;49;00m\r\n",
"\u001b[34mCOPY\u001b[39;49;00m tf_keras_resnet_byoc.py /opt/ml/code/tf_keras_resnet_byoc.py\r\n",
"\r\n",
"\u001b[37m# Defines train.py as script entrypoint\u001b[39;49;00m\r\n",
"\u001b[34mENV\u001b[39;49;00m SAGEMAKER_PROGRAM tf_keras_resnet_byoc.py\r\n"
]
}
],
"source": [
"! pygmentize docker/Dockerfile"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare a training script\n",
"\n",
"The following cell prints an example training script **tf_keras_resnet_byoc.py** in the **docker** folder. To register the Debugger hook, you need to use the Debugger client library `smdebug`. \n",
"\n",
"In the `main` function, a Keras hook is registered after the line where the `model` object is defined and before the line where the `model.compile()` function is called. \n",
"\n",
"In the `train` function, you pass the Keras hook and set it as a Keras callback for the `model.fit()` function. The `hook.save_scalar()` method is used to save scalar parameters for mini batch settings, such as epoch, batch size, and the number of steps per epoch in training and validation modes."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[33m\"\"\"\u001b[39;49;00m\r\n",
"\u001b[33mThis script is a ResNet training script which uses Tensorflow's Keras interface, and provides an example of how to use SageMaker Debugger when you use your own custom container in SageMaker or your own script outside SageMaker.\u001b[39;49;00m\r\n",
"\u001b[33mIt has been orchestrated with SageMaker Debugger hooks to allow saving tensors during training.\u001b[39;49;00m\r\n",
"\u001b[33mThese hooks have been instrumented to read from a JSON configuration that SageMaker puts in the training container.\u001b[39;49;00m\r\n",
"\u001b[33mConfiguration provided to the SageMaker python SDK when creating a job will be passed on to the hook.\u001b[39;49;00m\r\n",
"\u001b[33mThis allows you to use the same script with different configurations across different runs.\u001b[39;49;00m\r\n",
"\u001b[33m\u001b[39;49;00m\r\n",
"\u001b[33mIf you use an official SageMaker Framework container (i.e. AWS Deep Learning Container), you do not have to orchestrate your script as below. Hooks are automatically added in those environments. This experience is called a \"zero script change\". For more information, see https://github.com/awslabs/sagemaker-debugger/blob/master/docs/sagemaker.md#zero-script-change. An example of the same is provided at https://github.com/awslabs/amazon-sagemaker-examples/sagemaker-debugger/tensorflow2/tensorflow2_zero_code_change.\u001b[39;49;00m\r\n",
"\u001b[33m\"\"\"\u001b[39;49;00m\r\n",
"\r\n",
"\u001b[37m# Standard Library\u001b[39;49;00m\r\n",
"\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36margparse\u001b[39;49;00m\r\n",
"\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mrandom\u001b[39;49;00m\r\n",
"\r\n",
"\u001b[37m# Third Party\u001b[39;49;00m\r\n",
"\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mnumpy\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnp\u001b[39;49;00m\r\n",
"\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mcompat\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mv2\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mtf\u001b[39;49;00m\r\n",
"\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mkeras\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mapplications\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mresnet50\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m ResNet50\r\n",
"\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mkeras\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mdatasets\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m cifar10\r\n",
"\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtensorflow\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mkeras\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mutils\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m to_categorical\r\n",
"\r\n",
"\u001b[37m# smdebug modification: Import smdebug support for Tensorflow\u001b[39;49;00m\r\n",
"\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msmdebug\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mtensorflow\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36msmd\u001b[39;49;00m\r\n",
"\r\n",
"\r\n",
"\u001b[34mdef\u001b[39;49;00m \u001b[32mtrain\u001b[39;49;00m(batch_size, epoch, model, hook):\r\n",
" (X_train, y_train), (X_valid, y_valid) = cifar10.load_data()\r\n",
"\r\n",
" Y_train = to_categorical(y_train, \u001b[34m10\u001b[39;49;00m)\r\n",
" Y_valid = to_categorical(y_valid, \u001b[34m10\u001b[39;49;00m)\r\n",
"\r\n",
" X_train = X_train.astype(\u001b[33m'\u001b[39;49;00m\u001b[33mfloat32\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\r\n",
" X_valid = X_valid.astype(\u001b[33m'\u001b[39;49;00m\u001b[33mfloat32\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\r\n",
"\r\n",
" mean_image = np.mean(X_train, axis=\u001b[34m0\u001b[39;49;00m)\r\n",
" X_train -= mean_image\r\n",
" X_valid -= mean_image\r\n",
" X_train /= \u001b[34m128.\u001b[39;49;00m\r\n",
" X_valid /= \u001b[34m128.\u001b[39;49;00m\r\n",
" \r\n",
" \u001b[37m# register hook to save the following scalar values\u001b[39;49;00m\r\n",
" hook.save_scalar(\u001b[33m\"\u001b[39;49;00m\u001b[33mepoch\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, epoch)\r\n",
" hook.save_scalar(\u001b[33m\"\u001b[39;49;00m\u001b[33mbatch_size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, batch_size)\r\n",
" hook.save_scalar(\u001b[33m\"\u001b[39;49;00m\u001b[33mtrain_steps_per_epoch\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mlen\u001b[39;49;00m(X_train)/batch_size)\r\n",
" hook.save_scalar(\u001b[33m\"\u001b[39;49;00m\u001b[33mvalid_steps_per_epoch\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mlen\u001b[39;49;00m(X_valid)/batch_size)\r\n",
" \r\n",
" model.fit(X_train, Y_train,\r\n",
" batch_size=batch_size,\r\n",
" epochs=epoch,\r\n",
" validation_data=(X_valid, Y_valid),\r\n",
" shuffle=\u001b[34mFalse\u001b[39;49;00m,\r\n",
" \u001b[37m# smdebug modification: Pass the hook as a Keras callback\u001b[39;49;00m\r\n",
" callbacks=[hook])\r\n",
"\r\n",
"\r\n",
"\u001b[34mdef\u001b[39;49;00m \u001b[32mmain\u001b[39;49;00m():\r\n",
" parser = argparse.ArgumentParser(description=\u001b[33m\"\u001b[39;49;00m\u001b[33mTrain resnet50 cifar10\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n",
" parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--batch_size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m50\u001b[39;49;00m)\r\n",
" parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--epoch\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m15\u001b[39;49;00m)\r\n",
" parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model_dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[33m\"\u001b[39;49;00m\u001b[33m./model_keras_resnet\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\r\n",
" parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--lr\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m, default=\u001b[34m0.001\u001b[39;49;00m)\r\n",
" parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--random_seed\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mbool\u001b[39;49;00m, default=\u001b[34mFalse\u001b[39;49;00m)\r\n",
" \r\n",
" args = parser.parse_args()\r\n",
"\r\n",
" \u001b[34mif\u001b[39;49;00m args.random_seed:\r\n",
" tf.random.set_seed(\u001b[34m2\u001b[39;49;00m)\r\n",
" np.random.seed(\u001b[34m2\u001b[39;49;00m)\r\n",
" random.seed(\u001b[34m12\u001b[39;49;00m)\r\n",
"\r\n",
" \r\n",
" mirrored_strategy = tf.distribute.MirroredStrategy()\r\n",
" \u001b[34mwith\u001b[39;49;00m mirrored_strategy.scope():\r\n",
" \r\n",
" model = ResNet50(weights=\u001b[34mNone\u001b[39;49;00m, input_shape=(\u001b[34m32\u001b[39;49;00m,\u001b[34m32\u001b[39;49;00m,\u001b[34m3\u001b[39;49;00m), classes=\u001b[34m10\u001b[39;49;00m)\r\n",
"\r\n",
" \u001b[37m# smdebug modification:\u001b[39;49;00m\r\n",
" \u001b[37m# Create hook from the configuration provided through sagemaker python sdk.\u001b[39;49;00m\r\n",
" \u001b[37m# This configuration is provided in the form of a JSON file.\u001b[39;49;00m\r\n",
" \u001b[37m# Default JSON configuration file:\u001b[39;49;00m\r\n",
" \u001b[37m# {\u001b[39;49;00m\r\n",
" \u001b[37m# \"LocalPath\": \u001b[39;49;00m\r\n",
" \u001b[37m# }\"\u001b[39;49;00m\r\n",
" \u001b[37m# Alternatively, you could pass custom debugger configuration (using DebuggerHookConfig)\u001b[39;49;00m\r\n",
" \u001b[37m# through SageMaker Estimator. For more information, https://github.com/aws/sagemaker-python-sdk/blob/master/doc/amazon_sagemaker_debugger.rst\u001b[39;49;00m\r\n",
" hook = smd.KerasHook.create_from_json_file()\r\n",
"\r\n",
" opt = tf.keras.optimizers.Adam(learning_rate=args.lr)\r\n",
" model.compile(loss=\u001b[33m'\u001b[39;49;00m\u001b[33mcategorical_crossentropy\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m,\r\n",
" optimizer=opt,\r\n",
" metrics=[\u001b[33m'\u001b[39;49;00m\u001b[33maccuracy\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m])\r\n",
"\r\n",
" \u001b[37m# start the training.\u001b[39;49;00m\r\n",
" train(args.batch_size, args.epoch, model, hook)\r\n",
"\r\n",
"\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\r\n",
" main()\r\n"
]
}
],
"source": [
"! pygmentize docker/tf_keras_resnet_byoc.py"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Create a Docker image, build the Docker training container, and push to Amazon ECR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create a Docker image\n",
"\n",
"AWS Boto3 Python SDK provides tools to automatically locate your region and account information to create a Docker image uri."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import boto3\n",
"\n",
"account_id = boto3.client(\"sts\").get_caller_identity().get(\"Account\")\n",
"ecr_repository = \"sagemaker-debugger-mnist-byoc-tf2\"\n",
"tag = \":latest\"\n",
"\n",
"region = boto3.session.Session().region_name\n",
"\n",
"uri_suffix = \"amazonaws.com\"\n",
"if region in [\"cn-north-1\", \"cn-northwest-1\"]:\n",
" uri_suffix = \"amazonaws.com.cn\"\n",
"byoc_image_uri = \"{}.dkr.ecr.{}.{}/{}\".format(account_id, region, uri_suffix, ecr_repository + tag)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Print the image URI address."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"byoc_image_uri"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Optional Step] Login to access the Deep Learning Containers image repository\n",
"\n",
"If you use one of the AWS Deep Learning Container base images, uncomment the following cell and execute to login to the image repository."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ! aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-east-1.amazonaws.com"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Build the Docker container and push it to Amazon ECR\n",
"\n",
"The following code cell builds a Docker container based on the Dockerfile, create an Amazon ECR repository, and push the container to the ECR repository."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!docker build -t $ecr_repository docker\n",
"!$(aws ecr get-login --region $region --registry-ids $account_id --no-include-email)\n",
"!aws ecr create-repository --repository-name $ecr_repository\n",
"!docker tag {ecr_repository + tag} $byoc_image_uri\n",
"!docker push $byoc_image_uri"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note:** If this returns a permission error, see the [Get Started with Custom Training Containers](https://docs.aws.amazon.com/sagemaker/latest/dg/build-container-to-train-script-get-started.html#byoc-training-step5) in the Amazon SageMaker developer guide. Follow the note in Step 5 to register the **AmazonEC2ContainerRegistryFullAccess** policy to your IAM role."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Use Amazon SageMaker to set the Debugger hook and rule configuration"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define Debugger hook configuration\n",
"\n",
"Now you have the custom training container with the Debugger hooks registered to your training script. In this section, you import the SageMaker Debugger API operations, `Debugger hook Config` and `CollectionConfig`, to define the hook configuration. You can choose Debugger pre-configured tensor collections, adjust `save_interval` parameters, or configure custom collections.\n",
"\n",
"In the following notebook cell, the `hook_config` object is configured with the pre-configured tensor collections, `losses`. This will save the tensor outputs to the default S3 bucket. At the end of this notebook, we will retrieve the `loss` values to plot the overfitting problem that the example training job will be experiencing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import sagemaker\n",
"from sagemaker.debugger import DebuggerHookConfig, CollectionConfig\n",
"\n",
"sagemaker_session = sagemaker.Session()\n",
"\n",
"train_save_interval = 100\n",
"eval_save_interval = 10\n",
"\n",
"hook_config = DebuggerHookConfig(\n",
" collection_configs=[\n",
" CollectionConfig(\n",
" name=\"losses\",\n",
" parameters={\n",
" \"train.save_interval\": str(train_save_interval),\n",
" \"eval.save_interval\": str(eval_save_interval),\n",
" },\n",
" )\n",
" ]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Select Debugger built-in rules\n",
"\n",
"The following cell shows how to directly use the Debugger built-in rules. The maximum number of rules you can run in parallel is 20."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sagemaker.debugger import Rule, rule_configs\n",
"\n",
"rules = [\n",
" Rule.sagemaker(rule_configs.vanishing_gradient()),\n",
" Rule.sagemaker(rule_configs.overfit()),\n",
" Rule.sagemaker(rule_configs.overtraining()),\n",
" Rule.sagemaker(rule_configs.saturated_activation()),\n",
" Rule.sagemaker(rule_configs.weight_update_ratio()),\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5. Define a SageMaker Estimator object with Debugger and initiate a training job\n",
"\n",
"Construct a SageMaker Estimator using the image URI of the custom training container you created in **Step 3**.\n",
"\n",
"**Note:** This example uses the SageMaker Python SDK v1. If you want to use the SageMaker Python SDK v2, you need to change the parameter names. You can find the SageMaker Estimator parameters at [Get Started with Custom Training Containers](https://docs.aws.amazon.com/sagemaker/latest/dg/build-container-to-train-script-get-started.html#byoc-training-step5) in the AWS SageMaker Developer Guide or at [the SageMaker Estimator API](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html) in one of the older version of SageMaker Python SDK documentation."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sagemaker.estimator import Estimator\n",
"from sagemaker import get_execution_role\n",
"\n",
"role = get_execution_role()\n",
"\n",
"estimator = Estimator(\n",
" image_name=byoc_image_uri,\n",
" role=role,\n",
" train_instance_count=1,\n",
" train_instance_type=\"ml.p3.16xlarge\",\n",
" # Debugger-specific parameters\n",
" rules=rules,\n",
" debugger_hook_config=hook_config,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Initiate the training job in the background\n",
"\n",
"With the `wait=False` option, the `estimator.fit()` function will run the training job in the background. You can proceed to the next cells. If you want to see logs in real time, go to the [CloudWatch console](https://console.aws.amazon.com/cloudwatch/home), choose **Log Groups** in the left navigation pane, and choose **/aws/sagemaker/TrainingJobs** for training job logs and **/aws/sagemaker/ProcessingJobs** for Debugger rule job logs."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"estimator.fit(wait=False)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Print the training job name\n",
"\n",
"The following cell outputs the training job running in the background."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"job_name = estimator.latest_training_job.name\n",
"print(\"Training job name: {}\".format(job_name))\n",
"\n",
"client = estimator.sagemaker_session.sagemaker_client\n",
"\n",
"description = client.describe_training_job(TrainingJobName=job_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Output the current job status\n",
"\n",
"The following cell tracks the status of training job until the `SecondaryStatus` changes to `Training`. While training, Debugger collects output tensors from the training job and monitors the training job with the rules. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"if description[\"TrainingJobStatus\"] != \"Completed\":\n",
" while description[\"SecondaryStatus\"] not in {\"Training\", \"Completed\"}:\n",
" description = client.describe_training_job(TrainingJobName=job_name)\n",
" primary_status = description[\"TrainingJobStatus\"]\n",
" secondary_status = description[\"SecondaryStatus\"]\n",
" print(\n",
" \"Current job status: [PrimaryStatus: {}, SecondaryStatus: {}]\".format(\n",
" primary_status, secondary_status\n",
" )\n",
" )\n",
" time.sleep(15)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Retrieve output tensors using the smdebug trials class\n",
"\n",
"### Call the latest Debugger artifact and create a smdebug trial"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following smdebug `trial` object calls the output tensors once they become available in the default S3 bucket. You can use the `estimator.latest_job_debugger_artifacts_path()` method to automatically detect the default S3 bucket that is currently being used while the training job is running. \n",
"\n",
"Once the tensors are available in the dafault S3 bucket, you can plot the loss curve in the next sections."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"from smdebug.trials import create_trial\n",
"\n",
"trial = create_trial(estimator.latest_job_debugger_artifacts_path())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note:** If you want to re-visit tensor data from a previous training job that has already done, you can retrieve them by specifying the exact S3 bucket location. The S3 bucket path is configured in a similar way to the following sample: `trial=\"s3://sagemaker-us-east-1-111122223333/sagemaker-debugger-mnist-byoc-tf2-2020-08-27-05-49-34-037/debug-output\"`. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Print the hyperparameter configuration saved as scalar values"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trial.tensor_names(regex=\"scalar\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Print the size of the `steps` list to check the training progress"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from smdebug.core.modes import ModeKeys\n",
"\n",
"len(trial.tensor(\"loss\").steps(mode=ModeKeys.TRAIN))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(trial.tensor(\"loss\").steps(mode=ModeKeys.EVAL))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 7: Analyze the training job using the smdebug `trial` methods and the Debugger rule job status"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Plot training and validation loss curves in real time"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following cell retrieves the `loss` tensor from training and evaluation mode and plots the loss curves. \n",
"\n",
"In this notebook example, the dataset was `cifar10` that divided into 50,000 32x32 color training images and 10,000 test images, labeled over 10 categories. (See the [TensorFlow Keras Datasets cifar10 load data documentation](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/cifar10/load_data) for more details.) In the Debugger configuration step (Step 4), the save interval was set to 100 for training mode and 10 for evaluation mode. Since the batch size is set to 100, there are 1,000 training steps and 200 validation steps in each epoch. \n",
"\n",
"The following cell includes scripts to call those mini batch parameters saved by `smdebug`, computes the average loss in each epoch, and renders the loss curve in a single plot. \n",
"\n",
"As the training job proceeds, you will be able to observe that the validation loss curve starts deviating from the training loss curve, which is a clear indication of overfitting problem."
]
},
{
"cell_type": "code",
"execution_count": 400,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAEJCAYAAAAJqCSsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXhU5d3/8fc3e0IgkBVICGHJQtgXQZRVUdFHRUVb96VaW1sram2tv/bR1trWp4tFq9Zad2vVVlxaK1VUFlFU9n3fA4QkhiUkZL9/f8yAAZIhQCaTZD6v68o1M2fOnHwJkPM597kXc84hIiIiwSsk0AWIiIhIYCkMiIiIBDmFARERkSCnMCAiIhLkFAZERESCnMKAiIhIkPNbGDCzbmY208xWmdlKM5tSzz5mZo+Z2QYzW2ZmQ/xVj4iIiNQvzI/HrgZ+6JxbZGbtgYVmNsM5t6rOPucDmd6vEcCfvY8iIiLSTPwWBpxzu4Bd3uclZrYaSAXqhoFJwEvOM/PR52bW0cy6eD9br8TERJeRkeGvskVE2qSFCxcWOeeSAl2HtEz+bBk4zMwygMHAF0e9lQpsr/M6z7vtiDBgZrcCtwKkp6ezYMECf5UqItImmdnWQNcgLZffOxCaWSwwDbjTObf/ZI7hnHvaOTfMOTcsKUnBVkREpCn5NQyYWTieIPCKc+7NenbZAXSr8zrNu01ERESaiT9HExjwLLDaOfdIA7v9C7jeO6rgdGCfr/4CIiIi0vT82WfgTOA6YLmZLfFu+39AOoBz7ingPeACYANQBtzkx3pEROQULVy4MDksLOwZoB+aq6a1qAVWVFdX3zJ06NCC+nbw52iCuYAdZx8HfN9fNYiISNMKCwt7pnPnzn2SkpL2hISEuEDXI8dXW1trhYWFufn5+c8AF9e3j1KdiIiciH5JSUn7FQRaj5CQEJeUlLQPT2tO/fs0Yz0iItL6hSgItD7ev7MGz/lBEwbW7S7hl++uoqK6JtCliIiItChBEwby9pTx7NzNfLm5ONCliIjIScrPzw/NycnJzcnJyU1MTByYnJw84NDr8vJyn/3U5syZE3PjjTd287UPwODBg3OaotZ33323/fjx43s3xbH8rVlmIGwJRvZMJCIshJlrChmdqYmLRERao86dO9esWbNmFcDdd9/dNTY2tubBBx/cfej9qqoqwsPD6/3smDFjysaMGVN2vO+xePHiNU1WcCsRNC0D0RGhjOyZwKy19Y6qEBGRVmry5MkZV199dfqAAQNybrvttrSZM2fGDBo0KKdPnz65gwcPzlm6dGkkHHmlfvfdd3e94oorMoYPH56dlpbW/6GHHko+dLyYmJjBh/YfPnx49sSJE3v26NGj78UXX9yjtrYWgNdffz2uR48effv27dvnxhtv7HYiLQB/+ctf4rOysnIzMzP73nbbbakA1dXVTJ48OSMzM7NvVlZW7i9+8YtkgIceeii5V69efbOysnIvvPDCnk32QztK0LQMAIzPTuLn/17F1q9K6Z7QLtDliIi0aj96Y2m3dfklMU15zKzO7ct+d/nA7cff80i7du2KWLRo0ZqwsDCKi4tD5s+fvyY8PJy33367/Y9//OO0999/f+PRn9mwYUPUZ599tnbv3r2hffr06fejH/2oMDIy8ojOkatXr45esmTJpoyMjKqhQ4fmzJgxI3b06NGlU6ZM6T5r1qw1OTk5lRdddFGPxta5ZcuW8J///OepCxcuXJ2UlFQ9evTorJdffrljRkZG5a5du8LXr1+/EqCoqCgU4LHHHuu8devW5dHR0e7QNn8ImpYBgHHZnuA3a21hgCsREZGmdNlll+0JC/Nc3xYXF4decMEFvTIzM/v++Mc/7rZu3bqo+j5z7rnn7o2OjnZdunSpjo+Pr8rLyzvmArl///6lvXr1qgoNDaVv375lGzdujFiyZElUt27dKnJycioBrrzyykZ3Rps7d267008/vaRr167V4eHhfPOb3yyePXt2bE5OTsX27dsjb7jhhm5vvPFGh06dOtUAZGdnH7z00kt7PPnkk/Hh4eF+G8URVC0DGYnt6JnYjplrC7jhjIxAlyMi0qqdzBW8v8TGxtYeen7vvfemjh07tmTGjBkb165dG3HWWWdl1/eZuq0AoaGhVFdXH9MBsTH7NIWkpKSaFStWrHrrrbc6PPXUU0mvv/56/D//+c8tM2fOXD99+vT277zzTtzvf//7LmvXrl3ZUJ+IUxFULQMAY7OTmLfxKw5WaoihiEhbtH///tC0tLRKgL/85S+JTX38AQMGlG/fvj1y7dq1EQCvv/56fGM/O3r06NIvvvii/a5du8Kqq6v55z//GT9u3LgDu3btCqupqeHGG2/c+5vf/GbH8uXLY2pqati4cWPERRddVPLEE0/sOHDgQOi+ffv8cqsgqFoGAMZnJ/P8p1v4fNNXjM9JPv4HRESkVbn33nvzb7nllh7/93//1/Wcc87Z29THj42NdY888sjWiRMnZsbExNQOHDiwtKF9582b1yElJWXAodevvPLKxgceeGDH2LFjs5xzNmHChL3XXnvt3nnz5kXffPPNGbW1tQbw4IMP5lVXV9vVV1/do6SkJNQ5Z7fccktBYmKiX65kzbM8QOsxbNgwt2DBgpP+fHlVDYMfnMEVw9J4cFKDMzOKiLQpZrbQOTfsVI+zdOnSLQMHDixqippas3379oXExcXV1tbWcv3116dnZmaWP/DAAy16uNrSpUsTBw4cmFHfe0F3myAqPJQzeyfw8ZoCWlsQEhGRlmHq1KmJOTk5uZmZmX33798fevfdd7fqgBR0twnAM6rgw9UFbCwspXdybKDLERGRVuaBBx4oaOktASci6FoGAMZle2Yg1AREIiIiQRoG0jrFkJUSq/kGRERE8GMYMLPnzKzAzFY08H6cmf3bzJaa2Uozu8lftdRnXHYyX2z+itKK6ub8tiIiIi2OP1sGXgAm+nj/+8Aq59xAYBzwBzOL8GM9RxiXnURVjePTDa26z4eIiMgp81sYcM7NAXxN0eiA9mZmQKx332a7TB/WPZ7YyDBm6laBiEirMWLEiKxp06Z1qLvtwQcfTL7mmmvSG/rM8OHDs+fMmRMDMHbs2N71zfF/9913d73//vtTfH3vl19+uePChQsPT2185513dn377bfbn/if4kgtYanjQPYZeBzoA+wElgNTnHO19e1oZrea2QIzW1BY2DQn74iwEEb1TmTWWg0xFBFpLa644oriV1999YgZ/6ZNmxZ/7bXXNmp9gNmzZ2842Yl73n777Y7Lli2LPvR66tSpOy+55JKSkzlWSxPIMHAesAToCgwCHjezDvXt6Jx72jk3zDk3LCkpqckKGJ+TxK595azd3Sb+LkVE2rzrrrtuz8cffxxXXl5uAGvXro0oKCgIP++88w5cc8016f369evTu3fvvnfddVfX+j6fmpraf9euXWEA9957b+eMjIx+Q4cOzV6/fn3koX3+8Ic/JPbr169PdnZ27nnnnderpKQkZMaMGe0+/PDDjj/72c/ScnJycleuXBk5efLkjOeff74TwDvvvNO+T58+uVlZWblXXHFFxsGDB+3Q97vrrru65ubm9snKyspdvHhxvYsm1ac5lzoO5DwDNwEPO89l+QYz2wzkAF82VwGHVjGcuaaQnM715hAREWnI29/vRsGqJl3CmOTcMi55osEFkFJSUmoGDhxY+sYbb8Rde+21e1988cX4iy66aE9ISAiPPPLIjpSUlJrq6mrOOOOM7C+++CJ6xIgRB+s7zieffBLz1ltvxS9fvnxVVVUVgwYNyh08eHAZwDXXXLPnhz/8YRHAHXfc0fWxxx5L/OlPf1owYcKEvRdeeOG+m266aU/dY5WVldl3vvOdHh988MHaAQMGVFx66aUZv/vd75Luv//+AoDExMTqVatWrX744YeTHn744ZTXX3996/F+DM291HEgWwa2AWcDmFkKkA1sas4CUjpEkdulg+YbEBFpRb7xjW8Uv/76650A3nzzzfjrrruuGODFF1+Mz83N7ZObm5u7fv36qKVLlzZ4FT5z5szYCy64YG/79u1r4+Pja88999zDaxgsXLgweujQodlZWVm506ZNS1i5cqXPq/mlS5dGpaWlVQwYMKAC4MYbb/xq7ty5h/sSXH311XsAhg8fXrZ9+/bIho5TV3Mvdey3lgEzexXPKIFEM8sDHgDCAZxzTwG/BF4ws+WAAfc655q9a//4nCSemr2J/eVVdIhq+mUhRUTaLB9X8P509dVX7/3pT3/abe7cuTHl5eUho0ePLluzZk3E448/nuK9kq6ZPHlyRnl5+Uld8N5666093njjjQ0jR448+NhjjyXMnj37lDoJRkVFOYCwsDB3qksg+2upY3+OJrjKOdfFORfunEtzzj3rnHvKGwRwzu10zp3rnOvvnOvnnPubv2rxZVx2MjW1jrnrNcRQRKQ1iIuLqx05cmTJLbfcknHppZcWA+zZsyc0Ojq6Nj4+vmb79u1hs2bNivN1jLPOOuvAe++91/HAgQO2Z8+ekBkzZnQ89F5ZWVlIenp6VUVFhb322muHOyvGxsbW7N+//5jz5sCBA8t37NgRsWLFikiAl156KWH06NGn1BmtuZc6Dsq1Ceoa3K0jHaLCmLmmgAv6dwl0OSIi0ghXXnll8fXXX9/r1Vdf3QQwcuTIg/369Svr1atXvy5dulQOHTr0gK/Pjxo1quzSSy8t7tevX9+EhISqAQMGHF6G+Cc/+cnO4cOH94mPj68eMmTIgQMHDoQCXHPNNcW33XZbxlNPPZXyxhtvbDy0f0xMjHvqqae2XHHFFb1qamoYOHBg2T333HNCQ98CvdRx0C1hXJ/b/76ILzYX88V9ZxMSckotOCIiLZKWMBYtYXwc47OTKSypYNWu/YEuRUREpNkpDABjvasYzlyjUQUiIhJ8FAaAxNhIBqbFMVNDDEVEjqf20P1qaT28f2f1zvILCgOHjctOZsn2vewprQx0KSIiLdmKwsLCOAWC1qO2ttYKCwvjgHpXEQaNJjhsXHYSj360njnrC5k0KDXQ5YiItEjV1dW35OfnP5Ofn98PXVC2FrXAiurq6lsa2kFhwGtAWkfi20Uwa63CgIhIQ4YOHVoAXBzoOqRpKdV5hYYYY7OSmL2ukJra1jXcUkRE5FQoDNQxLjuJ4tJKluXtPf7OIiIibYTCQB1jMpMIMZi59oQmjhIREWnVFAbq6NQugsHpnbSKoYiIBBWFgaOMz05iWd4+CksqAl2KiIhIs1AYOMq47GQA5qzTrQIREQkOCgNHye3SgaT2kZqNUEREgobCwFFCQoxxWUnMWVdIdU2DMzeKiIi0GX4LA2b2nJkVmFmD0x+a2TgzW2JmK81str9qOVHjc5LZX17N4u0aYigiIm2fP1sGXgAmNvSmmXUEngQuds71Ba7wYy0nZFRmIqEhplUMRUQkKPgtDDjn5gDFPna5GnjTObfNu3+LOfN2iApnWPdOmm9ARESCQiD7DGQBncxslpktNLPrG9rRzG41swVmtqCwsHlO0ONzklm9az/5+8qb5fuJiIgESiDDQBgwFPgf4Dzgf80sq74dnXNPO+eGOeeGJSUlNUtx471DDGevazENFiIiIn4RyDCQB7zvnCt1zhUBc4CBAaznCFkpsXSJi2LmGt0qEBGRti2QYeAdYJSZhZlZDDACWB3Aeo5gZozLTmbuhiIqqzXEUERE2i5/Di18FZgHZJtZnpndbGbfNbPvAjjnVgP/BZYBXwLPOOcaHIYYCOOzkzhQUc2Crb76QYqIiLRuYf46sHPuqkbs8zvgd/6q4VSd2TuR8FBj1tpCzuiVGOhyRERE/EIzEPrQLjKMET0SNN+AiIi0aQoDxzEuO4n1BQfYXlwW6FJERET8QmHgOMbneIYYztIqhiIi0kYpDBxHz8R2dIuPZrZWMRQRkTZKYeA4zIzx2cl8uuEryqtqAl2OiIhIk1MYaITx2ckcrKrhy80aYigiIm2PwkAjnN4zgciwEGbqVoGIiLRBCgONEB0RysheCczSKoYiItIGKQw00vjsZDYXlbK5qDTQpYiIiDQphYFGOrSK4SzdKhARkTZGYaCR0hNi6JnYTrcKRESkzVEYOAHjspOZt+krDlZqiKGIiLQdCgMnYHxOEpXVtczbVBToUkRERJqMwsAJGN4jnujwUGau0a0CERFpOxQGTkBkWChn9k5k5toCnHOBLkdERKRJKAycoPE5SeTtOcjGwgOBLkVERKRJ+C0MmNlzZlZgZiuOs99pZlZtZpf7q5amNM47xFC3CkREpK3wZ8vAC8BEXzuYWSjwf8AHfqyjSaV2jCYrJVZTE4uISJvhtzDgnJsDHG9lnx8A04DmObM20X3+8dnJzN9SzIGK6iY5noiISCAFrM+AmaUClwJ/bsS+t5rZAjNbUFh4ks3z+cvh2XNh/86T+3wd47KTqapxfLpBQwxFRKT1C2QHwqnAvc652uPt6Jx72jk3zDk3LCkp6eS+W8UBKFgNz02EPVtO7hhewzI6ERsZpqmJRUSkTQhkGBgGvGZmW4DLgSfN7BK/fbfuI+GGd6B8Hzx/ARStP+lDhYeGMDozkZlrCjXEUEREWr2AhQHnXA/nXIZzLgN4A/iec+5tv37T1KFw43+gphKePx/yfQ508Gl8djL5+8tZk1/ShAWKiIg0P38OLXwVmAdkm1memd1sZt81s+/663s2Sud+cON7EBIOL/wP7Fh4UocZm52EGTw7d3MTFygiItK8wvx1YOfcVSew743+qqNeSVnwrenw4sXw4iS45p+e2wgnIKVDFLeN7cWTszYyrHsnrhye7qdiRURE/Ct4ZyDslAE3TYf2neHlS2HjzBM+xA/PzWZU70Tu/9dKluXtbfoaRUREmkHwhgGAuFS46T2I7wl//yasnX5CHw8NMR67ajBJsZHc9rdFFJdW+qlQERER/wnuMAAQmww3vgspfeH1a2HFmyf08fh2Efz52iEUllRwx6uLqanV6AIREWldFAYAYuLh+ncg7TSYdjMsfuWEPj4grSMPTurL3A1F/OGDtX4qUkRExD8UBg6J6gDXToMeY+Gd78GXfz2hj185PJ0rT+vGk7M28v7KfD8VKSIi0vQUBuqKaAdXvQbZF8B798Cnj57Qx39+cV8GpMVxzz+WsklLHIuISCuhMHC08Cj4xkvQ9zKYcT/M/E2jFziKCg/lyWuGEBZqfOflhZRqISMREWkFFAbqExoOk5+BQdfA7Idhxv82OhCkdYrhT1cNYWPhAe6dtkzTFYuISIunMNCQkFC4+HE47dvw2Z/gPz+E2uOuqQTAqMxE7jkvm3eX7dIMhSLSNJyDGrU2in/4bQbCNiEkBC74HUTEePoPVB2Ei/8Eocf/sd02thdLtu3lN9PX0D81jhE9E5qhYBFpM2qqYfcK2P4FbJsH276A026GMfcEujJpgxQGjscMJvwCImJh5q+gqgwu+yuERRznY8YfvjGQSY9/yvf/vpj/3DGKlA5RzVS0iLQ6FSWQN99z0t82z7NuSqW3I3KHNM+U6Sn9AlujtFkKA41hBmN/DOEx8MFPobocrnjR09nQh/ZR4fzluqFMeuJTbvvbQl67dSQRYbozIyLAvjzY9vnXV/67V4KrBQvxTII28CpIP93zFZcW6GqljbPW1sFt2LBhbsGCBYErYP6z8J+7PfMRXPWqZzjicby7bCe3/30xN4zszi8mKdmLNJuqg1D2FZQVex4PFnuf133tfb+mEmISPJOQxSRATKL3MQHaJXz9PCYBwqNPrI7aGs/JfvsXngCw7XPYn+d5L7wdpA2D9JGQPgJSh3nmPWliZrbQOTesyQ8sbYJaBk7UaTd7Wgje+R68fJknEMTE+/zIhQO6smTbXp6Zu5lB6R25dLBSvsgJcw7K90FJPpTsgtKiI0/mR5zc93geqw82fLzIOO+JP94zLXlohOc4BWugrMjznAYulsLbfR0c2iUeGRQOfYVFwc7FsP1z2D4fKks8n23f1XPST78Duo3wNP03oh+SiD/pX+DJGHSV58pg2s3wpyEw9l4YdrPPfgQ/OT+H5Tv2cd+by8lO6UBu16ZP/iKtVsWBr0/yvh4bOrlHdfz65NwhFVL6f32ij0mA6Pgjr/qjO3mGEPtSW+MJH6VF3oDxlTckeMNH3e1F6z2PlUdPNmaeJv8B3/j6yj+um+fWo0gLotsEpyJ/OXzwM9g0y7Py4YRfQJ+LGvyPXlhSwYV/+oTIsFD+ffso4mKO88tIpC1wDnYtgT1bGjjR50PF/mM/FxYNHbpA+y6epcbrPsameL5i4j1BoKVcWVeVe1onSougshRSciEqLtBVAbpNIL75LQyY2XPAhUCBc+6YG+Vmdg1wL2BACXCbc27p8Y7bosIAeH7RrZ/hmZiocA2knwHnPgRpQ+vdfeHWPVz59DxG9U7k2RtOIyREVwjSRtXWwrrp8MkfPD3jDwmNOPbkXt9jZAddQTchhQHxxZ9hYAxwAHipgTBwBrDaObfHzM4Hfu6cG3G847a4MHBITTUsfglm/hpKC6Hf5XD2/dCp+zG7vjxvC//7zkrumpDFlAmZzV+riD/VVMPKN+GTR6BwNXTKgDPu8PSKb9/F00Svk3yzUxgQX/zWtuacm2NmGT7e/6zOy8+B1t2rLjQMhn0L+l8Bc6fCvMdh9b/h9O/C6B8e0VR47endWbx9L1M/WseAbnGMz04OYOEiTaS6Apb8HT6d6rklkNTHMydH38taTjO+iNSrUS0DZtYOOOicqzWzLCAHmO6cqzrO5zKAd+trGThqv3uAHOfcLQ28fytwK0B6evrQrVu3HrfmgNu3Az7+JSx9zXNfc+xPYNhNhzstHays4bI/f8aOPWW8+4PRpCfEBLhgkZNUWQoLX/BM212yC7oOhtH3eFb/DNG8Gi2FWgbEl8aGgYXAaKAT8CkwH6h0zl1znM9lcJwwYGbjgSeBUc65r45XS4u9TdCQnUs8nQy3fAIJveGcBz2/JM3Y9lUZFz0+l64do3nztjOIjggNdLUijXdwD3z5DHz+pKfTXMZoTytYz3G6DdACKQyIL42N7eacKwMuA550zl0B9D3Vb25mA4BngEmNCQKtUtdBcMO/4arXAIPXroYXLoSdi0lPiGHqlYNYk7+fn769XCscSutwoABmPAB/7A8zH4Juw+FbH8CN70Kv8QoCIq1QY2/kmZmNBK4BbvZuO6XLWDNLB94ErnPOrTuVY7V4ZpB9PvSeAItehJm/gafHwYBvMv7s+5lydiZTP1zP4G4duW5kRqCrFanf3u3w2WOw6CVP/4C+l8Lou6Fz/0BXJiKnqLFh4E7gPuAt59xKM+sJzPT1ATN7FRgHJJpZHvAAEA7gnHsKuB9IAJ40z5VEdZtvwgoNh9Nu8XYy/CPMexJWvcOUEd9jXdZYHnx3FbldOzC0u+8ZDUWaVdF6T6fYZa95Xg+8Es68CxJ7B7YuEWkyJzy00MxCgFjnXD2zhPhfq+sz4MvebfDRL2H5P6iNSeSPVZN5sWIsv7xsEJMGpQa6Ogl2u5bB3Edg5dsQFglDboAzfgAduwW6MjkJ6jMgvjS2A+Hfge8CNXg6D3YAHnXO/c6/5R2rTYWBQ3Ys8nQy3PopBSHJvFIxmor+V3Hn5LOIClenQmlm+Svgo1/A+g88E/+cdguc/j2ITQp0ZXIKFAbEl8aGgSXOuUHeWQOHAD8BFjrnBvi7wKO1yTAAnpkM106n9sunsU2zcA4Whw8kdfytdB5xuefKTMSfKkth9v/BZ497Vs0b+X047dsQ3THQlUkTUBgQXxrbZyDczMKBS4DHnXNVZqau703JDHIuICTnAti7ja0fPk3XFa/Secb3qJx1HxFDroLB16qzlvjH+hmepbn3boPB13mGwB5nNU4RaTsaO7TwL8AWoB0wx8y6AwHpMxAUOqaTcflD2JRl/LLTr3m/vA/VXz4LT42Cv4yF+c/Awb2BrlLagpJ8+OeN8MrlniV3b3wPJj2uICASZE56bQIzC3POVTdxPcfVZm8TNKC6ppapH67nlVmL+XbcAm5uN5fIr1Z7fnH3uRiGXAfdR2mmNzkxtbWw8Dn48BeeYYJjfgRn3qHbUW2YbhOIL43tMxCHZ2jgGO+m2cCDzrl9fqytXsEWBg6Zs66Qu15fwsGqah4fF8JZZe/D8jegYp9nIZjB18LAqyFOoxDkOHavhH9Pgbz50GMMXDgVEnoFuirxM4UB8aWxYWAasAJ40bvpOmCgc+4yP9ZWr2ANAwC795dzx6uL+WJzMd8YlsYvzu9F9Mb3PJPAbPkELAR6ne1pLcg6H8IiAl2ytCSVZZ4OgvMe9yycdd6vYcA3NWNgkFAYEF9OaDTB8bY1h2AOA+C5bfDoR+t5fOYGMpNjeeLqIWSmtIfiTbD4Fc+qcSU7ISYBBlzpmSUuNtnTOzyyA4RoqGJQOqKD4LVwzi/VLyDIKAyIL40NA/OAHznn5npfnwn83jk30s/1HSPYw8Ahn6wv5M7XllBWWcNDl/Rj8lDvCtC1NbDxY09rwdrpUHvUwpIRsZ6rwsgOXweEqA71bIv7+rHufhHt1T+hNSnJh//eByvfhMQsuPCPkDEq0FVJACgMiC+NDQMDgZeAOO+mPcANzrllfqytXgoDX6t72+DyoWk8OKkvMRF1RouWFsHWz6B8H1Tsh/L9Xz+W7z12W8V+qKn0/U0tFJL7QOoQSB0KXYdAcq7Wq29pamth4fPeDoLlMOYeOHOKOggGMYUB8eWERhOYWQcA59x+M7vTOTfVb5U1QGHgSNU1tTz20Xr+NHMDvZNiefIa722Dk1VV7g0H+7wBYd+RgaGsCPKXw46FniVsAcKiocsATzhIHepZzz6+p+5FB8rulfDvOyHvS08Hwf/5o9YREIUB8elUhhZuc86lN3E9x6UwUL+564u48/XFlFbU8OCkvlwxzM/zxzsHezZ7plLescgTDnYtheqDnvejOh7ZepA6FNqn+LemQKmpgoqSr78qD3if7/c+Hqjznvd1RDtPX452yZ5pftsle18nQXSnkwtS6iAoPigMiC+nEga2O+eafcUShYGGFewv547XFvP5pmImD0njl5ccdZB8Sr0AABn+SURBVNvA32qqoXD11+FgxyIoWAWuxvN+h9QjA0LXwZ5+CEerra1zQi056sR6vG0lnmbxkFDPLY2QME8fh8PPvY8WUud5qPd56FH7eV+beabqPXxi33/kSb+6vHE/n4hYiGwP4TGez5YWff2zqSskzBMK2iU1HBgObY+J99S5/kNvB8Gt6iAo9VIYEF/UMtDG1NQ6HvtoPY99vJ5e3tsGWady2+BUVZZB/rI6AWGhp0UBAIPETE/nxCOurEsad+zwdp6T69FfYVGek2xttSdYHH7ufXS1dZ7X2e/wa+/Xofec81zJR7b31BoZ+/X3ioj1bmt/5PbIDl+f/CPbez5/9EiO2lrPrZbSAjhQAKWF3scCOFB47PajO4OCJ9REx3tu3yRkwkVT1UFQ6qUwIL74DANmVgLUt4MB0c65Zu81pjDQOJ9uKGLKa0soKa/ivvNzuH5kBiEhLaS5uKwYdi6CHYs9j9XlR55EjznB17Mton1wdVp0ztPp81BIKC08MjAk9IIR31UHQWmQwoD4ctItA4GiMNB4BSXl3PvGMmauLWRU70R+e/kAunaMDnRZIhIACgPii98GjJvZc2ZWYGYrGnjfzOwxM9tgZsvMbIi/aglWye2jeO7G0/jNZf1ZtG0P502dw1uL82htAVBERPzLn7PHvABM9PH++UCm9+tW4M9+rCVomRlXDU9n+pTRZKe0567Xl/K9VxZRXHqc+QRERCRo+C0MOOfmAMU+dpkEvOQ8Pgc6mlkXf9UT7LontOP174zk3ok5fLh6N+f+cQ4frd4d6LJERKQFCOS8sqnA9jqv87zbjmFmt5rZAjNbUFhY2CzFtUWhIcZt43rxr9tHkRgbwc0vLuAn05ZxoKLZV6IWEZEWpFVMMu+ce9o5N8w5NywpKSnQ5bR6fbp04J3bz+S2cb34x4LtnP/oHL7c7KsRR0RE2rJAhoEdQN1Ji9K826QZRIaFcu/EHP7xnZEYxjefnsdv3ltNRXU9k+CIiEibFsgw8C/geu+ogtOBfc65XQGsJygNy4hn+pTRXDU8nb/M2cTFf/qUlTv3BbosERFpRv4cWvgqMA/INrM8M7vZzL5rZt/17vIesAnYAPwV+J6/ahHf2kWG8etL+/P8TadRXFbJJU98yhMzN1BdUxvo0kREpBlo0iE5wp7SSn729gr+s3wXQ9I78sg3BpGR2C7QZYnIKdKkQ+JLq+hAKM2nU7sIHr96MI9eOYgNBQc4/9FP+NvnWzVRkYhIG6YwIMcwMyYNSuWDu8YyLKMTP3t7BTc+P5/d+xu5Op+IiLQqCgPSoM5xUbz0reH8clJfvtj8Fef+cQ7/Xroz0GWJiEgTUxgQn8yM60ZmMH3KGHoktuMHry7mume/YNXO/YEuTUREmojCgDRKj8R2vPHdkdx/YS7Ld+zjf/70Cff8cym79h0MdGkiInKKNJpATti+siqenLWB5z/dQkgI3DKqJ98Z25P2UeGBLk1EGqDRBOKLwoCctO3FZfz+g7W8s2QnCe0iuHNCJlcOTyc8VA1OIi2NwoD4ot/actK6xcfw6JWD+dftZ5KZEsv/vrOS86bO4YOV+RqKKCLSiigMyCkbkNaRV799Os9cPwwDbn15Id/8y+cs2b430KWJiEgjKAxIkzAzJuSm8P6dY3jokn5sKjrAJU98yg9eXcz24rJAlyciIj6oz4D4xYGKap6evZGnP9lEbS1cP7I7t5/Vm44xEYEuTSQoqc+A+KIwIH6Vv6+cR2as5Z8L8+gQFc7t43tz/RndiQwLDXRpIkFFYUB80W0C8avOcVH89vKBTJ8ymkHdOvKr91Yz4ZHZ/GvpTnUyFBFpIRQGpFnkdO7Ai98azss3Dyc2Mpw7Xl3MJU9+xpebiwNdmohI0NNtAml2NbWOtxbv4A8frGXXvnJG9Ihn8tA0zu/XWRMXifiJbhOILwoDEjDlVTW8+NkWXpu/nc1FpUSFh3Be385cNiSNM3slEKbJi0SajMKA+OLXMGBmE4FHgVDgGefcw0e9nw68CHT07vMT59x7vo6pMND2OOdYvH0vby7K499Ld7HvYBVJ7SO5ZFBXLhuSRp8uHQJdokirpzAgvvgtDJhZKLAOOAfIA+YDVznnVtXZ52lgsXPuz2aWC7znnMvwdVyFgbatorqGmWsKeXNRHjPXFlBV4+jTpQOTh6Ry8aCuJLePCnSJIq2SwoD4EubHYw8HNjjnNgGY2WvAJGBVnX0ccOiyLw7Y6cd6pBWIDAtlYr/OTOzXmeLSSt5dtpNpi3bw0H9W8+v3VjMmK4lLB6dybm5noiM0PFFEpCn4s2XgcmCic+4W7+vrgBHOudvr7NMF+ADoBLQDJjjnFtZzrFuBWwHS09OHbt261S81S8u1oeAAby3O461FO9i5r5zYyDAu6O/pXzA8I56QEAt0iSItmloGxJdAh4G7vTX8wcxGAs8C/ZxztQ0dV7cJglttreOLzcW8uSiP95bvorSyhtSO0Vw2JJVLB6fSMyk20CWKtEgKA+KLP28T7AC61Xmd5t1W183ARADn3DwziwISgQI/1iWtWEiIMbJXAiN7JfDgpH58sCqfaYt28MTMDfzp4w0M6taRb57WjclD0ogI02gEEZHG8GfLQBieDoRn4wkB84GrnXMr6+wzHXjdOfeCmfUBPgJSnY+i1DIg9dm9v5x3luxg2sIdrN1dQmrHaH5wVm8mD00jXEMURdQyID75e2jhBcBUPMMGn3PO/crMHgQWOOf+5R1B8FcgFk9nwh875z7wdUyFAfHFOcec9UU88sFalubto3tCDHeclcmkQV01b4EENYUB8UWTDkmb5Jzjo9UFPDJjHat27adnUjumnJ3JRQO6qrOhBCWFAfFFl0rSJpkZE3JTePcHo3jq2iGEh4Qw5bUlTHx0DtOX76K2tnWFYBERf1IYkDYtJMSY2K8L06eM5rGrBlNd67jtlUVc+Ke5zFi1WysnioigMCBBIiTEuHhgVz64cwyPfGMgpZXVfPulBVzyxKfMWlugUCAiQU19BiQoVdXU8uaiPB77aAM79h5kaPdO3H1OFmf0SsBMfQqk7VGfAfFFYUCCWmV1Lf9YsJ3HP95A/n7Pcso/PDeb4T3iA12aSJNSGBBfFAZE8Cyn/NqX23hi1kYKSyoYnZnIXedkMSS9U6BLE2kSCgPii8KASB0HK2v42+db+fPsjRSXVjI+O4m7zsliQFrHQJcmckoUBsQXhQGRepRWVPPivC08PWcTe8uqyEyOZWK/zpzXtzN9u3ZQvwJpdRQGxBeFAREfSsqrmLYwj/+uzOfLzcXUOkjtGM15fTtzXt8UhmXEE6pJjKQVUBgQXxQGRBqpuLSSD1fv5v0V+XyyoYjK6loS2kVwTm4K5/XtzBm9E4gMCw10mSL1UhgQXxQGRE7CgYpqZq8t5L8r85m5poADFdXERoYxPieZ8/qmMC47mdhIfy4KKnJiFAbEF4UBkVNUUV3DZxu/4v0V+cxYtZuvSiuJCAthdO9EzuvbmQm5KcS3iwh0mRLkFAbEF4UBkSZUU+tYuHUP/12Rz/sr89mx9yAhBsN7xHv7GXSma8foQJcpQUhhQHxRGBDxE+ccK3fu5/2VnmCwbvcBAAakxTEuO5nRmYkM6taRcC2tLM1AYUB8URgQaSabCg/w/srdfLAqn6Xb91LrIDYyjNN7xnNm70RGZybSKylWwxbFLxQGxBeFAZEA2FdWxbxNRXyyvohPNxSx5asyADp3iGJUZiKjeidyZu9EktpHBrhSaSsUBsQXv4YBM5sIPAqEAs845x6uZ59vAD8HHLDUOXe1r2MqDEhbtL24jLkbipi7oYjPNhSxp6wKgJzO7RnVO5FRmYmM6JFAdISGLsrJURgQX/wWBswsFFgHnAPkAfOBq5xzq+rskwn8AzjLObfHzJKdcwW+jqswIG1dba2nr8EnGwr5dEMR87fsobK6lojQEIZ078jozCRG9U6kX2qcJjySRlMYEF/8GQZGAj93zp3nfX0fgHPuN3X2+S2wzjn3TGOPqzAgweZgZQ3ztxTz6QbPbYVVu/YDEBcdzhm9EhiVmcjYrCTSOsUEuFJpyRQGxBd/zoqSCmyv8zoPGHHUPlkAZvYpnlsJP3fO/ffoA5nZrcCtAOnp6X4pVqSlio4IZUxWEmOykrgPKDpQwacbPH0N5q4vYvqKfAD6p8YxsV9nJvbrTK+k2MAWLSKtij9bBi4HJjrnbvG+vg4Y4Zy7vc4+7wJVwDeANGAO0N85t7eh46plQORrzjk2FZXy4ardTF+Rz5Ltnv86WSmxTOzbmYn9utCnS3uNUBC1DIhP/mwZ2AF0q/M6zbutrjzgC+dcFbDZzNYBmXj6F4jIcZgZvZJi6TU2lu+M7cWufQd5f0U+01fk8/jMDTz28Qa6J8R4g0FnBnXrqGAgIsfwZ8tAGJ4OhGfjCQHzgaudcyvr7DMRT6fCG8wsEVgMDHLOfdXQcdUyINI4RQcqmOFtMfhsQxHVtY4ucVGc5w0Gp2nFxaCilgHxxd9DCy8ApuLpD/Ccc+5XZvYgsMA59y/zXKL8AZgI1AC/cs695uuYCgMiJ25fWRUfrfEEgznrCqmoriUxNoJzcj3B4IxeCZoJsY1TGBBfNOmQSJAprahm1tpCpq/Yxcw1BZRW1tAhKowJuSlM7NuZMVlJRIVrPoO2RmFAfFEYEAli5VU1h0ckfLh6N/sOVhETEcq47CRG9kxgeI8EMpNjCdHthFZPYUB80YLrIkEsKjyUCbkpTMhNoaqmlnkbv+K/K/P5eHUB7y33DFnsGBPOaRnxjOgRz/Ae8eR26UCYbimItCkKAyICQHhoyOH5DNwljrw9B/liczFfbv6KLzcXM2PVbgDaRYQy1BsORvSIp39aHJFhuq0g0prpNoGINMru/eV8ubn48Nfa3SUARIaFMDi9I8N7JDCiRzyD0zsSE6HrjJZGtwnEF4UBETkpe0or+XLL1+Fg5c591DoICzH6p8Ux3NtyMLR7PHHR4YEuN+gpDIgvCgMi0iRKyqtYuHXP4XCwNG8vVTUOM8jp3IHu8TEkd4gkpUMUSe0jSW4fSXL7KJI7RBIfE6FOin6mMCC+qC1PRJpE+6hwxmUnMy47GfCMVFi8bS9fbi5m4bY9bCg8wGcbi9hfXn3MZ8NC7HBASPIGhBTvY93QkNAuQp0XRfxAYUBE/CIqPJSRvRIY2SvhiO3lVTUU7K+goKScgpIKCvZ7Hnd7t+XtKWPRtj0Ul1Yec8wQg4RYT0AY0SOBm0f3ILVjdHP9kUTaLIUBEWlWUeGhpCfEkJ7ge8nlyupaCg98HRYKSioo9D7fsfcgL83bwovztnDxwK7cOqYnfbp0aJb6RdoihQERaZEiwkJI7Rjd4JX/zr0HeXbuZl79chtvLd7BuOwkvjOmF6f3jNdiTCInSB0IRaRV21dWxcufb+GFz7ZQdKCSgWlxfHdsL87t21kLMdWhDoTii8KAiLQJ5VU1TFuUx9NzNrH1qzJ6JLbj26N7ctmQVK21gMKA+KYwICJtSk2t4/2V+Tw1eyPL8vaRGBvJTWdmcO2I7sTFBO98BwoD4ovCgIi0Sc455m36iqdmb2LOukLaRYRy1fB0vjWqB12DcASCwoD4ojAgIm3eqp37eXrORv69bBcGTBqUynfG9iQrpX2gS2s2CgPii8KAiASN7cVlPDt3M6/P387BqhrOzknmO2N7cVpGpzY/AkFhQHzxaxgws4nAo0Ao8Ixz7uEG9psMvAGc5pzzeaZXGBCRU7WntJKX5m3lxXlbKC6tZHB6Ry4dnIoBFdW1db5qqKiqpbKmlooqz+vKOu9V1tm38tD+3uchZlw0sAs3j+pJ7+TYQP+RFQbEJ7+FATMLBdYB5wB5wHzgKufcqqP2aw/8B4gAblcYEJHmcrCyhn8u3M5fP9nE9uKDx7wfGRZCRFgIkWGhRIaFfP06PJTI0BAiw0OO2SfCu19xaRXvLttJRXUtZ+ckc8vongGdA0FhQHzxZxgYCfzcOXee9/V9AM653xy131RgBvAj4B6FARFpbjW1jt37yw+fyCPCQogIDTnlE/dXByp4+fOtvDRvK8WllfRL7cC3R/fkgv5dCG/mNRYUBsQXf/5rTAW213md5912mJkNAbo55/7j60BmdquZLTCzBYWFhU1fqYgEtdAQo2vHaBJjI2kfFU5kWGiTXMEnxEZy54QsPvvJWfz60v6UVdQw5bUljP3tTJ75ZBMl5VVNUL3IqQvY8l9mFgI8AvzwePs65552zg1zzg1LSkryf3EiIk0oKjyUq0ek8+HdY3nm+mGkxcfw0H9Wc8ZvPubX761m595jb1GINCd/rk2wA+hW53Wad9sh7YF+wCxvAu8M/MvMLj7erQIRkdYoJMSYkJvChNwUluXt5a+fbObZuZt5bu5mLhzQhVtG96Rfalygy5Qg5M8+A2F4OhCejScEzAeuds6tbGD/WajPgIgEmbw9ZTz/6RZe+3IbpZU1jOyZwK1jejI2K4mQJlxbQX0GxBe/3SZwzlUDtwPvA6uBfzjnVprZg2Z2sb++r4hIa5LWKYb/vTCXz+47m/vOz2FzUSk3vTCfc6fO4bUvt1FeVRPoEiUIaNIhEZEWpLK6lv8s38nTczazetd+EmMjuH5kBtee3p34dhEnfVy1DIgvCgMiIi2Qc47PNn7FXz/ZxKy1hUSFh3DPudncMrrnSR1PYUB88WcHQhEROUlmxpm9EzmzdyLrdpfwzCebSA3CBZakeSgMiIi0cFkp7fnt5QMDXYa0YQGbZ0BERERaBoUBERGRIKcwICIiEuQUBkRERIKcwoCIiEiQUxgQEREJcgoDIiIiQU5hQEREJMi1uumIzawQ2HqSH08EipqwHH9QjaeupdcHLb/Gll4ftPwaW1p93Z1zSYEuQlqmVhcGToWZLWjpc3OrxlPX0uuDll9jS68PWn6NLb0+kbp0m0BERCTIKQyIiIgEuWALA08HuoBGUI2nrqXXBy2/xpZeH7T8Glt6fSKHBVWfARERETlWsLUMiIiIyFEUBkRERIJc0IQBM5toZmvNbIOZ/STQ9RzNzLqZ2UwzW2VmK81sSqBrqo+ZhZrZYjN7N9C11MfMOprZG2a2xsxWm9nIQNdUl5nd5f37XWFmr5pZVAuo6TkzKzCzFXW2xZvZDDNb733s1AJr/J3373mZmb1lZh1bUn113vuhmTkzSwxEbSKNERRhwMxCgSeA84Fc4Cozyw1sVceoBn7onMsFTge+3wJrBJgCrA50ET48CvzXOZcDDKQF1WpmqcAdwDDnXD8gFLgysFUB8AIw8ahtPwE+cs5lAh95XwfSCxxb4wygn3NuALAOuK+5i6rjBY6tDzPrBpwLbGvugkRORFCEAWA4sME5t8k5Vwm8BkwKcE1HcM7tcs4t8j4vwXMSSw1sVUcyszTgf4BnAl1LfcwsDhgDPAvgnKt0zu0NbFXHCAOizSwMiAF2BrgenHNzgOKjNk8CXvQ+fxG4pFmLOkp9NTrnPnDOVXtffg6kNXthX9dS388Q4I/AjwH11JYWLVjCQCqwvc7rPFrYibYuM8sABgNfBLaSY0zF84utNtCFNKAHUAg8772V8YyZtQt0UYc453YAv8dzlbgL2Oec+yCwVTUoxTm3y/s8H0gJZDGN8C1geqCLqMvMJgE7nHNLA12LyPEESxhoNcwsFpgG3Omc2x/oeg4xswuBAufcwkDX4kMYMAT4s3NuMFBK4Ju3D/Ped5+EJ7R0BdqZ2bWBrer4nGf8cYu9sjWzn+K5zfZKoGs5xMxigP8H3B/oWkQaI1jCwA6gW53Xad5tLYqZheMJAq84594MdD1HORO42My24LnNcpaZ/S2wJR0jD8hzzh1qUXkDTzhoKSYAm51zhc65KuBN4IwA19SQ3WbWBcD7WBDgeuplZjcCFwLXuJY1aUovPKFvqff/TBqwyMw6B7QqkQYESxiYD2SaWQ8zi8DTaetfAa7pCGZmeO51r3bOPRLoeo7mnLvPOZfmnMvA8/P72DnXoq5qnXP5wHYzy/ZuOhtYFcCSjrYNON3MYrx/32fTgjo4HuVfwA3e5zcA7wSwlnqZ2UQ8t60uds6VBbqeupxzy51zyc65DO//mTxgiPffqEiLExRhwNvJ6HbgfTy/fP/hnFsZ2KqOcSZwHZ4r7iXerwsCXVQr9APgFTNbBgwCfh3geg7ztli8ASwCluP5/xfwKWvN7FVgHpBtZnlmdjPwMHCOma3H06LxcAus8XGgPTDD+//lqRZWn0iroemIRUREglxQtAyIiIhIwxQGREREgpzCgIiISJBTGBAREQlyCgMiIiJBTmFA5ChmVlNneOeSplzl0swy6lvZTkQkkMICXYBIC3TQOTco0EWIiDQXtQyINJKZbTGz35rZcjP70sx6e7dnmNnHZrbMzD4ys3Tv9hQze8vMlnq/Dk09HGpmfzWzlWb2gZlFB+wPJSKCwoBIfaKPuk3wzTrv7XPO9ccz+91U77Y/AS865wbgWSznMe/2x4DZzrmBeNZIODTrZSbwhHOuL7AXmOznP4+IiE+agVDkKGZ2wDkXW8/2LcBZzrlN3kWl8p1zCWZWBHRxzlV5t+9yziWaWSGQ5pyrqHOMDGCGcy7T+/peINw595D//2QiIvVTy4DIiXENPD8RFXWe16C+OyISYAoDIifmm3Ue53mff4ZnJUeAa4BPvM8/Am4DMLNQM4trriJFRE6ErkhEjhVtZkvqvP6vc+7Q8MJO3hURK4CrvNt+ADxvZj8CCoGbvNunAE97V7CrwRMMdvm9ehGRE6Q+AyKN5O0zMMw5VxToWkREmpJuE4iIiAQ5tQyIiIgEObUMiIiIBDmFARERkSCnMCAiIhLkFAZERESCnMKAiIhIkPv/s4gY7soA3ZYAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"# Retrieve the loss tensors collected in training mode\n",
"y = []\n",
"for step in trial.tensor(\"loss\").steps(mode=ModeKeys.TRAIN):\n",
" y.append(trial.tensor(\"loss\").value(step, mode=ModeKeys.TRAIN)[0])\n",
"y = np.asarray(y)\n",
"\n",
"# Retrieve the loss tensors collected in evaluation mode\n",
"y_val = []\n",
"for step in trial.tensor(\"loss\").steps(mode=ModeKeys.EVAL):\n",
" y_val.append(trial.tensor(\"loss\").value(step, mode=ModeKeys.EVAL)[0])\n",
"y_val = np.asarray(y_val)\n",
"\n",
"train_save_points = int(\n",
" trial.tensor(\"scalar/train_steps_per_epoch\").value(0)[0] / train_save_interval\n",
")\n",
"val_save_points = int(trial.tensor(\"scalar/valid_steps_per_epoch\").value(0)[0] / eval_save_interval)\n",
"\n",
"y_mean = []\n",
"x_epoch = []\n",
"for e in range(int(trial.tensor(\"scalar/epoch\").value(0)[0])):\n",
" ei = e * train_save_points\n",
" ef = (e + 1) * train_save_points - 1\n",
" y_mean.append(np.mean(y[ei:ef]))\n",
" x_epoch.append(e)\n",
"\n",
"y_val_mean = []\n",
"for e in range(int(trial.tensor(\"scalar/epoch\").value(0)[0])):\n",
" ei = e * val_save_points\n",
" ef = (e + 1) * val_save_points - 1\n",
" y_val_mean.append(np.mean(y_val[ei:ef]))\n",
"\n",
"plt.plot(x_epoch, y_mean, label=\"Training Loss\")\n",
"plt.plot(x_epoch, y_val_mean, label=\"Validation Loss\")\n",
"\n",
"plt.legend(bbox_to_anchor=(1.04, 1), loc=\"upper left\")\n",
"plt.xlabel(\"Epoch\")\n",
"plt.ylabel(\"Loss\")\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check the rule job summary\n",
"\n",
"The following cell returns the Debugger rule job summary. In this example notebook, we used the five built-in rules: `VanishingGradient`, `Overfit`, `Overtraining`, `SaturationActivation`, and `WeightUpdateRatio`. For more information about what each of the rules evaluate on the on-going training job, see the [List of Debugger built-in rules](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-built-in-rules.html) documentation in the Amazon SageMaker developer guide. Define the following `rule_status` object to retrieve Debugger rule job summaries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rule_status = estimator.latest_training_job.rule_job_summary()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the following cells, you can print the Debugger rule job summaries and the latest logs. The outputs are in the following format:\n",
"\n",
"```\n",
"{'RuleConfigurationName': 'Overfit',\n",
" 'RuleEvaluationJobArn': 'arn:aws:sagemaker:us-east-1:111122223333:processing-job/sagemaker-debugger-mnist-b-overfit-e841d0bf',\n",
" 'RuleEvaluationStatus': 'IssuesFound',\n",
" 'StatusDetails': 'RuleEvaluationConditionMet: Evaluation of the rule Overfit at step 7200 resulted in the condition being met\\n',\n",
" 'LastModifiedTime': datetime.datetime(2020, 8, 27, 18, 17, 4, 789000, tzinfo=tzlocal())}\n",
"```\n",
"\n",
"The `Overfit` rule job summary above is an actual output example of the training job in this notebook. It changes `RuleEvaluationStatus` to the `IssuesFound` status when it reaches the global step 7200 (in the 6th epoch). The `Overfit` rule algorithm determines if the training job is having Overfit issue based on its criteria. The default criteria to invoke the overfitting issue is to have at least 10 percent deviation between the training loss and validation loss.\n",
"\n",
"Another issue that the training job has is the `WeightUpdateRatio` issue at the global step 500 in the first epoch, as shown in the following log.\n",
"\n",
"```\n",
"{'RuleConfigurationName': 'WeightUpdateRatio',\n",
" 'RuleEvaluationJobArn': 'arn:aws:sagemaker:us-east-1:111122223333:processing-job/sagemaker-debugger-mnist-b-weightupdateratio-e9c353fe',\n",
" 'RuleEvaluationStatus': 'IssuesFound',\n",
" 'StatusDetails': 'RuleEvaluationConditionMet: Evaluation of the rule WeightUpdateRatio at step 500 resulted in the condition being met\\n',\n",
" 'LastModifiedTime': datetime.datetime(2020, 8, 27, 18, 17, 4, 789000, tzinfo=tzlocal())}\n",
"```\n",
"\n",
"This rule monitors the weight update ratio between two consecutive global steps and determines if it is too small (less than 0.00000001) or too large (above 10). In other words, this rule can identify if the weight parameters are updated abnormally during the forward and backward pass in each step, not being able to start converging and improving the model.\n",
"\n",
"In combination of the two issues, it is clear that the model is not well setup to improve from the early stage of training. \n",
"\n",
"Run the following cells to track the rule job summaries.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**`VanishingGradient` rule job summary**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rule_status[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**`Overfit` rule job summary**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rule_status[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**`Overtraining` rule job summary**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rule_status[2]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**`SaturationActivation` rule job summary**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rule_status[3]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**`WeightUpdateRatio` rule job summary**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rule_status[4]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook Summary and Other Applications\n",
"\n",
"This notebook presented how you can have insights into training jobs by using SageMaker Debugger for any of your model running in a customized training container. The AWS cloud infrastructure, the SageMaker ecosystem, and the SageMaker Debugger tools make debugging process more convenient and transparent. The Debugger rule's `RuleEvaluationStatus` invocation system can be further extended to the Amazon CloudWatch Events and AWS Lambda function to take automatic actions, such as stopping training jobs once issues are detected. A sample notebook to set the combination of Debugger, CloudWatch, and Lambda is provided at [Amazon SageMaker Debugger - Reacting to CloudWatch Events from Rules](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-debugger/tensorflow_action_on_rule/tf-mnist-stop-training-job.ipynb)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "conda_python3",
"language": "python",
"name": "conda_python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.10"
}
},
"nbformat": 4,
"nbformat_minor": 4
}